top of page

Confident Isn't Correct: AMBOSS and MDCalc partner on the Case for Cross-Checking Clinical AI

Writer: MDCalc Team
MDCalc Team
6 days ago
3 min read

Ask a clinical AI tool a question and the answer usually comes back fast, well written and confident. That confidence is the problem. A language model sounds the same whether it's right or wrong, and it can make serious mistakes on the tasks clinicians assume are simplest, such as running a risk score.


In the latest episode of MD Aware: Upgrading Clinical Judgment, MDCalc co-founder and CEO Dr. Joe Habboushe talks with Dr. Madjid Salimi of AMBOSS about how to build clinical AI that admits what it doesn't know for certain, and why AMBOSS chose to be one of the first models in CrossCover, MDCalc's new platform for cross-checking clinical AI.


Listen to the Episode


Meet the Guest

Dr. Madjid Salimi is a physician by training and the co-founder and co-CEO of AMBOSS, the medical knowledge platform founded by a team of doctors in Berlin in 2012. AMBOSS is now used by more than a million clinicians and students worldwide. Its AI product, AMBOSS AI, answers clinical questions from the AMBOSS library of physician-written, editorially reviewed content.

Host Dr. Joe Habboushe is an emergency physician and the co-founder and CEO of MDCalc.


Built to Say "I'm Not Sure"

AMBOSS has used AI for years in search, in sorting user feedback, and in helping editors track guideline changes. When large language models arrived, Madjid saw the speed of innovation he had been hoping for. He also saw the risk. Hundreds of thousands of physicians use AMBOSS every day, and he says the company can't release something that can sometimes make mistakes.

So AMBOSS AI doesn't write answers on its own. It has to cite AMBOSS's hand-selected sources and the clinical guidelines its team has approved. And it is designed to show conflicts and uncertainty, even when that makes an answer look less impressive. "You don't want this mentor who's blindly sending you in the wrong way," Madjid says.


What NOHARM Found

That cautionary approach proved itself when it showed up in NOHARM, a benchmark from researchers at Stanford and Harvard that tested AI tools on real clinical cases and counted recommendations that could cause severe harm. AMBOSS AI had the lowest rate of any system tested, at 2.9%. Madjid points out that completeness matters too, because leaving out a key step can also hurt a patient.

Joe took an additional lesson from the results. Even the best tools still make potentially severe errors, and the study was only looking for the most obvious, serious mistakes. Subtler errors that mislead without being plainly wrong are harder to catch. Clinical AI is helpful, Joe says, but it's still a long way from something clinicians can rely on without double checking.


A Layer Between the Clinician and the Model

That gap is what CrossCover is built to close. Clinicians can put a question to the clinical AI model of their choice,  starting with AMBOSS AI and Vera Health. Before the question is sent, PHI Scrubbing removes patient identifiers, and TrackerBlocking removes details about who is asking. When the answer comes back, CrossCheck compares it with guidelines and the published literature, marking supported statements with a green check and conflicts with an amber flag. If the question involves a score, the validated MDCalc calculator appears next to the answer.

For partner models, joining means giving something up: they never learn which clinician is asking. Joe explains why that matters. In other industries, AI is already being used to set prices based on who the customer is. He calls the idea of that reaching clinical medicine "nauseating." Madjid sees the separation as a plus. "Every clinical AI should be open to that," he says. "Putting another layer on top of this will only help to make it better." He compares clinical AI today to the early internet, when companies like AOL tried to do everything inside their own walls, and he expects to see specialized tools working together will beat any one company trying to do it all.


The Takeaway

Joe closes with a saying at MDCalc: moving fast and breaking things may work in Silicon Valley, but not when the things breaking are patients. He leaves listeners with three habits. Don't let a language model do your arithmetic. Keep patient identifiers out of AI tools. And treat a confident answer as a claim to check, not a conclusion.


CrossCover is live now at crosscover.ai, with AMBOSS AI and Vera Health available at launch.


MD Aware: Upgrading Clinical Judgment is a podcast from MDCalc. Listen on Apple Podcasts and Spotify.

Don't Miss an Update!

Thanks for signing up!

Unsubscribe at any time.

  • LinkedIn
  • facebook
  • Bluesky
bottom of page