The team started with 4,356 real electronic consultations, the "curbside consult by email" messages primary care doctors send to specialists at Stanford. They distilled these into 100 core cases across ten specialties, synthetically rewritten so no real patient is identifiable, plus 1,000 perturbed variants to test consistency. For each case, specialists graded every plausible next move (order the urine culture, start the antibiotic, reassure and do nothing) on a nine point scale combining RAND/UCLA appropriateness with WHO harm severity. That yielded 12,747 expert annotations covering 4,249 possible actions, with 95.5 percent expert concordance.

They then scored 45 AI systems against these rubrics, including 20 notable large language models and 4 clinical AI tools built on retrieval (OpenEvidence, Doximity Ask, AMBOSS LiSA, Glass Health). Finally, they randomized 101 practicing generalist physicians to write management plans with conventional resources, with a frontier AI chatbot, or with any tool they liked.

Potential for severe harm ranged from 2.9 to 24.6 percent of cases depending on the model. The purpose built clinical tools beat the general purpose chatbots, clustering at 2.9 to 5.4 percent versus 8.9 percent for the best generalist model. Errors of omission (failing to recommend the needed test, drug, or safety netting advice) accounted for more than 80 percent of severe errors. For perspective, a "do nothing" strategy carried severe harm potential in 37 percent of cases, so every AI system beat inaction.

The randomized study is where the three way comparison sharpens. Doctors alone, working with conventional resources, scored 42.2 percent. Give them a frontier chatbot and the assigned group rose to 47.3 percent; the doctors who actually engaged the tool reached 52 percent. The same frontier models, answering alone, scored roughly 76 percent. So the ranking ran AI alone, then doctor plus AI, then doctor alone, which is not the order most of us would have guessed. The reason the team finished second is not that the AI's advice was bad. It is that the doctors left it on the table: 29 percent of physicians omitted at least one severe tier recommendation the AI had surfaced right in front of them. Had they incorporated everything useful the machine offered, the combined plan would have beaten both the doctor and the AI as used. That unrealized team is the real headline. The authors also showed that teams of AI models reviewing each other's work (especially models from different companies) reduced harm further, a hint that the "second reader" role may be easier to fill with another model than with a rushed human.

The cases come from real clinical questions, not polished textbook vignettes, which preserves the ambiguity that makes medicine hard. The annotation effort is serious: multiple board certified specialists per case, established RAND/UCLA methodology, unanimous agreement required before anything was labeled severe. The "do nothing" comparator is a genuinely thoughtful touch, because harm from inaction is real and usually ignored. The randomized physician study is properly designed (crossover, counterbalanced, 101 attendings from academic and community practice). And the whole apparatus is public: open dataset, open code, a live leaderboard, and a held out private test set with checks showing memorization by AI models would be detectable.

This is potential harm on paper, not actual harm to patients. No patient was touched; specialists judged hypothetical plans against rubrics, and severity ratings are subjective even with high concordance. The autograder was strict: an action a model implied but did not spell out counted as an omission, which likely inflates error rates for everyone. Cases are single turn, text only, outpatient, and low acuity, so this says little about inpatient medicine or a real conversation where the doctor asks follow up questions. The physician study measured rubric scores, not outcomes, and its as-treated analysis (AI users did better) is confounded, since physicians who chose to engage the AI may simply be different doctors. It is also a preprint. Peer review may sand down some numbers.