UpToDate Expert AI declined to answer nineteen of every hundred questions it was asked. The three frontier models declined between one and three. I have consulted with people at both ends of that range. Only one of them ever worried me, and it was not the one who occasionally said no.

That number sits in a figure panel most readers will skip, in a paper whose title does the heavy lifting. General purpose large language models outperform specialized clinical AI tools on medical benchmarks. Vishwanath and colleagues at NYU Langone and UT Austin put GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 against OpenEvidence and UpToDate Expert AI, threw in Google Search AI Overview as a control nobody asked for, and had twelve blinded clinicians grade 1,800 responses. The frontier models won everything.

I believe the result. I want to talk about what it measures, because the gap between "scored higher when a doctor read it" and "should be the thing a doctor uses" is where the entire clinical AI field currently lives.

Three stages. Five hundred USMLE style multiple choice questions from MedQA. Five hundred free response prompts from HealthBench, graded by a panel of three LLM judges. And the new one, the Real Clinical Queries benchmark, built from 100 de-identified questions that physicians actually typed into NYU Langone's internal HIPAA compliant GPT instance during ordinary work. Twelve US clinicians rated every response on clinical correctness, completeness, safety, and clarity, blinded to which system produced it, three raters per response.

This study will be read as another entry in the "AI beats doctors" file. It is not one. No physician was tested against a model here. Physicians were the graders. The comparison is between two kinds of software, and the specialized software lost.

That distinction matters because the actual literature on models versus clinicians is stranger and more useful than the headlines. Two studies should be stapled to this one.

Goh and colleagues ran a randomized trial with 50 physicians on diagnostic reasoning cases. Physicians given GPT-4 scored a median 76 percent. Physicians with conventional resources scored 74 percent. The difference was not significant (P = 0.60). The model working alone scored 16 points above the conventional group. The knowledge was there. Handing it to a doctor produced almost nothing.

Bean and colleagues at Oxford went further and tested the public. Across 1,298 participants and ten clinical scenarios, the models identified the correct condition 94.9 percent of the time when tested on their own. When actual humans used those same models to work through the same scenarios, correct identification fell below 34.5 percent, no better than people using whatever they would normally use. Users did not know what to tell the model. The model answered differently depending on trivial changes in phrasing. Correct and incorrect content arrived in the same paragraph, wearing the same tone.

Every number in the Nature Medicine paper is a measurement of a model alone. The Oxford work is the measurement of a model with a person attached, and the person is where sixty points of accuracy go to die. That is not an argument against the technology. It is an argument that benchmark rank order tells you almost nothing about deployed performance, which is, to be fair, roughly the argument the authors themselves are making about vendor benchmarks. They just stopped one step short of applying it to their own.

I have spent eight sections on the limits, so let me be clear that I think this is a good paper and a needed one. Four things in it are worth more than the headline.

The RCQ construction is the reusable asset. One hundred real questions from real clinicians in a live deployment, de-identified under an IRB protocol, uncontaminated by anyone's training data. Any health system running an internal LLM already has this raw material sitting in its logs. Building your own version of this benchmark is the highest yield evaluation work available to a CMIO right now, because it measures your clinicians asking your questions, and because no vendor can study for it.

The error taxonomy is more actionable than the scores. Of 153 categorized failures, incomplete clinical content accounted for 55 and safety critical omission for 33. Hallucination accounted for 4. The failure mode everyone builds governance around is the rarest one in the data. Omission is the real hazard, and it is far harder to catch, because nothing on the screen looks wrong. A confident, well organized, beautifully formatted answer that leaves out the contraindication reads exactly like a correct answer. Detection strategies built to catch fabrication will not catch this.

Retrieval is not automatically an upgrade. The authors point to evidence that retrieval augmented generation can degrade performance when irrelevant material gets pulled in or poorly integrated. This is a useful correction to the assumption that grounding a model in a curated corpus is strictly additive. It is not the retrieval that helps. It is the retrieval being good, which is a much harder engineering problem than buying a corpus and a vector database.

It is a procurement fact. A subscription product at roughly $699 a year and a free ad supported product scored in the same statistical tier as the AI summary that appears above your Google results, on questions your own clinicians actually asked. That does not mean cancel the subscription. UpToDate carries editorial accountability, medicolegal defensibility, and a citation trail that none of the frontier models match, and this study did not measure any of those. It does mean the burden of proof has moved. Ask your vendor for independent evaluation on real clinical queries. If they hand you their own benchmark, you have learned something.

All of which lands at an awkward moment. In January the FDA widened enforcement discretion so that AI clinical decision support offering a single recommendation can reach the market without review, provided a clinician can independently review the basis for it. This paper argues that independent, real world evaluation of clinical AI is urgently needed. The regulatory floor moved the other way in the same six months. Whatever evaluation happens now is going to happen inside health systems, or it is not going to happen.