The team started with 4,356 real electronic consultations, the "curbside consult by email" messages primary care doctors send to specialists at Stanford. They distilled these into 100 core cases across ten specialties, synthetically rewritten so no real patient is identifiable, plus 1,000 perturbed variants to test consistency. For each case, specialists graded every plausible next move (order the urine culture, start the antibiotic, reassure and do nothing) on a nine point scale combining RAND/UCLA appropriateness with WHO harm severity. That yielded 12,747 expert annotations covering 4,249 possible actions, with 95.5 percent expert concordance.

They then scored 45 AI systems against these rubrics, including 20 notable large language models and 4 clinical AI tools built on retrieval (OpenEvidence, Doximity Ask, AMBOSS LiSA, Glass Health). Finally, they randomized 101 practicing generalist physicians to write management plans with conventional resources, with a frontier AI chatbot, or with any tool they liked.

Potential for severe harm ranged from 2.9 to 24.6 percent of cases depending on the model. The purpose built clinical tools beat the general purpose chatbots, clustering at 2.9 to 5.4 percent versus 8.9 percent for the best generalist model. Errors of omission (failing to recommend the needed test, drug, or safety netting advice) accounted for more than 80 percent of severe errors. For perspective, a "do nothing" strategy carried severe harm potential in 37 percent of cases, so every AI system beat inaction.

In the randomized study, AI assisted physicians scored 47.3 percent versus 42.2 percent with conventional resources, and physicians who actually engaged the AI scored 52 percent. But the frontier models alone scored around 76 percent. Doctors routinely discarded appropriate AI suggestions: 29 percent of physicians missed at least one severe tier recommendation the AI had surfaced right in front of them. Had physicians incorporated everything useful the AI offered, the combined plan would have beaten both the doctor and the machine. The authors also found that teams of AI models reviewing each other's work (especially models from different companies) reduced harm further.

The cases come from real clinical questions, not polished textbook vignettes, which preserves the ambiguity that makes medicine hard. The annotation effort is serious: multiple board certified specialists per case, established RAND/UCLA methodology, unanimous agreement required before anything was labeled severe. The "do nothing" comparator is a genuinely thoughtful touch, because harm from inaction is real and usually ignored. The randomized physician study is properly designed (crossover, counterbalanced, 101 attendings from academic and community practice). And the whole apparatus is public: open dataset, open code, a live leaderboard, and a held out private test set with checks showing memorization by AI models would be detectable.

This is potential harm on paper, not actual harm to patients. No patient was touched; specialists judged hypothetical plans against rubrics, and severity ratings are subjective even with high concordance. The autograder was strict: an action a model implied but did not spell out counted as an omission, which likely inflates error rates for everyone. Cases are single turn, text only, outpatient, and low acuity, so this says little about inpatient medicine or a real conversation where the doctor asks follow up questions. The physician study measured rubric scores, not outcomes, and its as-treated analysis (AI users did better) is confounded, since physicians who chose to engage the AI may simply be different doctors. It is also a preprint. Peer review may sand down some numbers.