Know Which AI You’re Trusting
These essays are intended to spark discussion about where technology is taking clinical decision support – and where physicians feel it should go.
A rule, a regression, and a language model fail in different ways. Clinicians, health systems, and tool builders should know which one is talking.
Many people new to the field now use “AI” to mean generative AI. It’s one slice of something much larger. IBM defines AI as “technology that enables computers and machines to simulate human learning, comprehension, problem solving, decision making, creativity and autonomy.” The breadth is intentional, and it matters more than it sounds once AI starts advising clinicians. Different kinds of AI fail in different ways, and a clinician, health system, or builder who can’t tell them apart will have a harder time understanding their limitations. But knowing the type of AI is only the starting point. What matters just as much is what the system has been designed to do, how well it performs, and what happens when it makes a mistake.
Rules do exactly what they’re told
Rule-based AI follows prewritten, hard-coded rules and doesn’t learn from data. A simple clinical decision rule is a familiar example: a programmer specifies which inputs to collect, what calculations to perform, and how to interpret the result. Many tools on MDCalc work this way. They take clinical inputs and apply predetermined rules, equations, or published statistical models to produce a result. The software itself isn't learning from each patient, even when the underlying clinical model was originally derived from patient data.
Machine learning learns the rules instead
Machine learning uses models trained on data to recognize patterns and make predictions. It has three main learning approaches. In supervised learning, the model learns to predict an outcome from associated inputs. A spam filter predicts, from the text, whether a message is spam. In unsupervised learning, there’s no labeled outcome to train on. The model finds structure in the data, often by clustering similar data points. Apple Photos grouping pictures of the same person or pet illustrates this kind of pattern recognition. Reinforcement learning uses rewards and penalties to teach a model which actions achieve a goal. A computer learning to play a game through trial and error is a classic example.
These approaches aren't mutually exclusive, and real-world AI systems often combine them.
Deep learning is a size class, not a fourth branch
Machine learning has roots in the 1950s, although statistical techniques used in machine learning date back much further. Linear regression is a simple example of supervised learning: fit a line relating X inputs to a Y output. The math is modest. A neural network can learn far more intricate associations between inputs and outputs. Deep learning uses neural networks with multiple layers to learn increasingly complex patterns, often demanding substantially more computing power. A plant-identification app, for example, may need to process millions of image pixels rather than a handful of clinical variables. Deep learning is a separate axis from the three learning approaches above: deep architectures can be applied to supervised, unsupervised, or reinforcement learning alike.
Generative AI predicts the next word
Generative models produce new output (text, images, audio) rather than a label or a score. They draw on several of the branches above, but today’s popular tools use a newer architecture called the transformer. Large language models are the quintessential example. They’re trained on enormous amounts of text to take in a string and predict the next word.
Each type fails differently
AI is entering clinical decision support through tools like OpenEvidence and ChatGPT-style products built for healthcare. These run on LLMs and inherit LLM limitations. The signature one is hallucination: output that sounds or looks plausible and is wrong or fabricated. In text, that means invented facts and citations. In image models, it’s extra or missing fingers, lettering that isn’t real words, anatomy that’s subtly off in ways you only catch on close inspection. Either way, the model can generate convincing output without reliably distinguishing what is true from what merely looks plausible.
I saw this firsthand when I led a pilot of an ambient AI scribe. Because these tools are built on LLMs, I was watching specifically for hallucination, and it showed up in one of the ways that matters most. The scribe pended a prescription that was never part of the plan. It was a confident, plausible output with no basis in what actually happened during the visit. We did not roll the tool out beyond the pilot.
This illustrates why the consequences of an AI error matter as much as its frequency. A scribe might generate hundreds of accurate notes, but an invented medication order is a very different kind of failure from an awkward sentence or a minor documentation error. It's not enough to ask how often a model gets something wrong. We also need to ask what happens when it does, whether someone will catch the error, and what the system is allowed to do with its output.
Other model types fail too, through misclassification, miscalibration, and brittleness outside their training data. But they get checked differently. When I evaluate computer vision models for skin cancer classification, I look at sensitivity, specificity, and positive predictive value, because predictions can be compared against a reference diagnosis. Those numbers also have to be understood in the population where the model will actually be used. A model with impressive sensitivity might still generate large numbers of false positives, unnecessary biopsies, and referrals if deployed across a low-risk population.
Evaluating generative AI can be more complicated, particularly when the output is open-ended text with no single right answer to score against. But not every generative AI task is open-ended. A model extracting medications, summarizing clinical findings, or calculating a risk score can be evaluated against a reference standard. Developing quantitative metrics for more complex generative AI tasks in healthcare remains an active area of research.
That research also looks at omissions, where a model leaves out something clinically important, such as an allergy or a medication change mentioned during the visit. An omission can be harder to catch than a hallucination, because there is nothing wrong on the page to notice. And not all errors are equally important: leaving out a minor historical detail is very different from omitting a medication allergy or inventing a treatment recommendation.
Reason past the black box
Physicians spend years learning physiology and pathophysiology so they can reason when pattern recognition runs out: a presentation they’ve never seen quite this way, a case that doesn’t resemble any they’ve memorized. The same logic applies to AI. Understanding how each type works helps clinicians reason past a black-box answer instead of accepting it as truth. A fixed rule, a model trained on outcomes data, and a language model prone to inventing and omitting have different limitations, and they deserve different checks.
But understanding the underlying technology is only a starting point. A clinical prediction model might accurately predict which patients historically received a treatment without telling us which patients actually benefit from it. A language model might produce an excellent clinical summary while occasionally inventing a medication that was never discussed. And even a perfectly implemented clinical decision rule can be harmful if applied to the wrong population.
If you’re a clinician, health system administrator, or health tech builder, you need to understand what differentiates these types of AI before you use or implement the rapidly expanding range of health AI tools. But you also need to ask: What was this tool designed to do? How was it validated, and in whom? What mistakes does it make, how often, and what happens to patients when it gets something wrong? And how much autonomy does it have to act on its output?
Before you act on an AI recommendation, ask which type of model produced it.
Dr. Vamsi Varra is a board-certified dermatologist and CEO and co-founder of DermFlo, a smartphone app-based dermatology telehealth platform. He has a research background in machine learning applications in dermatology.