How Sick Are They?

These essays are intended to spark discussion about where technology is taking clinical decision support – and where physicians feel it should go.

Machine learning learns our habits, not our physiology. Counting abnormalities might be a better place to start.

You’re starting the first of a run of night shifts. You have a list of patients to see. Who goes first?

You want the ones where early intervention might mean a night on the ward instead of a night in intensive care. Location and the nature of the task matter too. But a key contributor is some measure of how unwell the person is, and your handover list doesn’t have a “how sick are they?” column.

The missing column

It might have an early warning score like NEWS. I like NEWS. It’s simple, effective and tells you something meaningful. But it’s a late sign. A patient bleeding internally can lose a lot of haemoglobin before their vital signs budge. A NEWS of zero leaves room for considerable variation in how unwell someone is.

You could turn to MDCalc and find hundreds of scores, but many are disease-specific, and most can’t be calculated until particular tests have been done. If one patient has a Glasgow-Blatchford score of 6 and another has a CURB-65 of 4, who is sicker? Do I really want to stick a needle in an 88-year-old’s radial artery just to calculate their APACHE or SOFA score? In my PhD, I set out to discover whether that missing column could be filled and, if so, how best to fill it.

At first, I was tempted by the siren call of machine learning. Many MDCalc scores come from an era when every datum was harvested by hand, by a research nurse armed with a biro and a clipboard, so you had to decide in advance what to collect. The electronic health record (EHR) is more like a battery farm, producing thousands of data points per patient. And patients rarely arrive with just one organ in trouble, so a single-organ score may miss the story. Why not pour the lot into a computer, press a button and see what comes out?

Patterns of recording, not patterns of physiology

The trouble is that EHRs record patterns of care, not patterns of physiology. What gets recorded depends on what someone, somewhere, decided needed recording. Missing data may be missing because nobody thought it mattered. The record is shaped by what a clinician has already suspected, tested for, diagnosed and treated. Machine learning models recognise patterns, and patterns of recording get mistaken for patterns of disease.

I call this the early warning paradox. Our bleeding patient has sprung a leak, and we have a good treatment: replace the blood, patch the hole. Contrast a frail 101-year-old with no obvious acute issue. Reaching 101 is remarkable, but staying alive does not get easier with practice. The bleeder has an identifiable focus of damage we can repair. The centenarian has accumulated widespread wear, and parts cannot be replaced.

Pour both patients’ data, and thousands more like them, into a computer, and it may learn that bleeders tend to survive while the oldest often do not. Use those predictions to decide who to see, and I might delay the bleeding patient to review centenarians. The model has learned from our success in treating bleeding, and its recommendation risks undoing it.

This extends beyond mortality. Train a model to “diagnose” and you may be training it to read the doctor’s mind. A clinician who suspects sarcoidosis orders a serum ACE, and the order itself becomes the signal. The original Epic Sepsis Model is the real-world version. Kamran and colleagues tested it before clinicians had shown signs of recognising sepsis themselves, such as giving antibiotics, ordering blood cultures or measuring lactate. Restricted to predictions before those actions, the model was no better than flipping a coin. Apparently, the best way to get it to tell you to consider sepsis was to start treating it yourself.

Then there’s the fact that medicine has been wrong time and time again, and the errors don’t come labelled in the training data. A patient in the early stages of variant CJD may be walking, talking and haemodynamically stable. NEWS zero, diagnosis terminal. A model trained on previous patients would predict death with great confidence. But what if there’s a drug in the pharmacy that could cure them and has never been tried? The previous patients would still have died, and the training data would look exactly the same. Our ignorance would have an impeccable evidence base. We want a “Move 37” for medicine. We’ve trained a model to predict the consequences of leaving the drug on the shelf.

A barometer, not a weather forecast

So I went back to the problem and asked clinicians what they wanted. One intensivist’s comment stayed with me: “If you tell me there’s an 80% chance my patient will die, that doesn’t mean I’m going to stop treating them. It makes me hope that what I’m doing puts them in the 20% who survive.”

That clarified what I wanted on my night shift: a barometer, not a weather forecast. How unwell is the patient now? Rather than asking a model to learn medicine from the electronic record, could we start with a system that already measures health and adapt it to EHR data? It didn’t need to capture everything. It needed to be a useful starting point.

The cumulative deficit model of frailty explains the difference between an 85-year-old who runs marathons and one who hasn’t left the house in a week. I saw no reason its principles should wait for someone’s 65th birthday, or stop applying when they become acutely ill. Frailty indices count the deficits present as a proportion of those assessed. We counted EHR data points that are abnormal, often labs but not exclusively. The index doesn’t depend on one fixed set of measurements, and a score can still be calculated with some data missing, provided enough suitable data are available.

Validated against outcomes, not defined by them

We published in PLOS Medicine in September. Despite its simplicity, the approach worked remarkably consistently. A score of 0.4 seemed to tell a similar story about someone’s health in a US emergency department, at King’s College Hospital or in UK Biobank, and across sexes and ethnicities. The patients and the measurements changed. Counting abnormalities still gave us a common language.

We called the measure FI-Lab, for its origins, but the arithmetic has no loyalty to the laboratory or the geriatric ward. Any measurement that can be classified as normal or abnormal can contribute.

An astute reader will point out that we assessed FI-Lab against outcomes. Are we really doing anything different from machine learning? The difference is that validation came at the end. We started with the principle of deficit counting and used outcomes to check whether it worked, not to fit weights to variables. FI-Lab is validated against outcomes, not defined by them.

That doesn’t prove it improves patient care. The best test is prospective: implement it in a hospital, randomise which patients have the score displayed in their records, and see whether the information changes anything. I’m working towards that.

Somewhere to start

Our findings suggest FI-Lab measures something meaningful, though any individual estimate carries considerable uncertainty. When a resident calls for advice, I want an early sense of whether we’re discussing someone severely unwell or someone who might go home. Both conversations begin with “Can I just ask you about a patient?” It helps to know which one we’re having. As I learn more, I use my own clinical experience to refine that estimate and narrow the uncertainty around it.

When we tested FI-Lab with clinicians alongside a state-of-the-art transformer model, they found it useful. Its consistency across populations suggests a common currency for these conversations, one that travels across diseases, hospitals and patient backgrounds. I began with a missing column on a handover list, but the same question turns up on ward rounds and at the other end of the telephone.

And it’s free. Our paper and implementation guidance explain which measurements to include, how many you need and when a result has passed its shelf life. You might find it simple. You might find it rough. I hope you’ll find it useful.

FI-Lab gives us somewhere to start.

About MDCalc

Since 2005, MDCalc has built clinical decision support around transparent evidence, physician judgment, and trust. We believe those same principles should guide the next generation of clinical AI.