Microsoft AI built a benchmark that turns published diagnostic cases into interactive encounters, and a system that works through those encounters by simulating a panel of five clinical roles. On 304 New England Journal of Medicine clinicopathological conference (CPC) cases the system, running on GPT‑5, reached 83.2% accuracy. On 325 emergency department admissions from MIMIC‑IV‑ED it reached 80.3%. Physicians given the same interface and allowed to use UpToDate scored 36.1% and 46.8%.

The preprint of this work appeared in June 2025 alongside a Microsoft blog post titled The Path to Medical Superintelligence, with physicians at 20% and the system four times better. The Science version adds an emergency department dataset, an open book physician arm, a judge sensitivity analysis and a much more careful discussion. A separate closed book group, with no outside resources, scored 22.3% on the CPC cases.

SDBench has three automated parts and one diagnostician. The diagnostician (a physician or a model) starts with a two or three sentence vignette and may, on each turn, ask history or exam questions, order named tests, or commit to one final diagnosis.

The CPC set is 304 consecutive Case Records of the Massachusetts General Hospital from 2017 to 2025. The 56 most recent (2024 to 2025) were held out from development and are the only CPC cases the physicians saw. The ED set is 325 MIMIC‑IV‑ED visits that ended in inpatient admission, with the reference diagnosis taken from the discharge summary. Seventy one ED cases, reviewed by three clinicians, were held out. All cases were screened for a clear presenting complaint, key findings and a definitive final diagnosis, so no case in either set is benign, undiagnosed or still ambiguous at discharge.

The supplement reports a processing error in the ED set. The diagnostic agent saw the patient's real age, but the Gatekeeper was given an age invented in an earlier version of the vignette. A stratified mixed effects check found no effect on accuracy or cost.

Early pilots told the diagnostician the information was unavailable when a question or test was absent from the case. That told the diagnostician which paths the case writers had taken. The fix was to have the Gatekeeper generate a realistic result for anything not documented, consistent with the rest of the case and not marked as synthetic. Pathognomonic findings are released only when the exact confirmatory test is ordered; vague requests are refused.

Synthetic answers dominate. In the primary GPT‑5 configuration 62.5% of Gatekeeper responses on CPC cases were invented (6,689 of 10,699), and 80.3% on ED cases (13,159 of 16,384), an average of 40.5 invented answers per ED case against 9.9 drawn from the record. For ED cases the Gatekeeper keeps a hidden ledger of what it invented, which the Judge sees so that a diagnosis built on an invented finding is not marked wrong.

Three checks for leakage. A physician panel reviewed 508 CPC responses and flagged eight, none judged to reveal the diagnosis. In a replay test the first three or five non synthetic answers were regenerated with the diagnosis removed or replaced by a wrong one; a GPT‑5 model diagnosing from only those few answers scored 46.3% with the diagnosis aware versions, 47.5% blind and 43.4% with a wrong diagnosis, with no significant difference. And accuracy fell as the share of invented answers rose (88.6% to 73.3% across CPC strata; point biserial r of minus 0.16 for CPC and a negligible minus 0.06 for ED). That last check is confounded, since thin records both require more invention and are harder to diagnose.

The rubric scores core disease, etiology, site, specificity and completeness, anchored on whether management would change. Physicians regraded all 56 held out CPC diagnoses from the best system and 56 physician diagnoses; agreement after binarizing was kappa 0.70 for the machine set and 0.87 for the human set, and in four of five disagreements the physicians thought the Judge too strict. A second check rescored 1,258 saved adjudications with GPT‑5.5 (kappa 0.89) and Claude Opus 4.8 (kappa 0.86). The physician validation covers CPC cases only. Where physicians and the Judge disagreed, the Judge was usually the stricter, which if anything lowers the reported accuracy.

Tests are mapped to CPT codes by a language model (98% matched; the rest priced by the model directly) and priced from the Kaiser Permanente Washington Central Hospital standard charges file, published under the CMS price transparency rule. Every run of questions counts as one $300 physician visit, which ends when a test is ordered. The number of questions inside a visit does not change the cost.

One model plays five roles. Dr. Hypothesis keeps a ranked top three differential and updates it after each finding. Dr. Test Chooser picks up to three discriminating tests per round. Dr. Challenger argues against the leading diagnosis. Dr. Stewardship vetoes low yield expensive tests. Dr. Checklist checks test names and internal consistency. Variants range from Instant Answer (vignette only) through Question Only and Budgeted to Unconstrained and an Ensemble of parallel panels. It was developed on GPT‑4.1 and the CPC validation cases, then run unchanged on GPT‑5 and later GPT‑5.5.

In role ablations on the CPC validation set, removing Dr. Stewardship cut accuracy from 84.7% to 81.9% and added $7,451 per case; removing Dr. Hypothesis and Dr. Stewardship together cut accuracy to 78.2%. The ablation p values are nominal and uncorrected across 25 configurations.

Three groups from the US or UK with a median of 12 years in practice. The main text describes the pool as including internal medicine specialists and acute and emergency care physicians; the supplement counts list only primary care physicians and general internists. The closed book CPC group (21 physicians: 17 primary care, 4 general internists) could use no outside resources. The open book CPC group (19 physicians: 8 primary care, 11 internists) could use UpToDate and NICE guidance but no generative AI. The ED group (16 physicians drawn from the first two pools) worked open book only. Physicians completed as many randomly ordered held out cases as they chose, with no time limit, one question or test per turn. The AI agents were allowed up to five questions and three tests per turn. A single turn ablation with o3 gave the same accuracy as batched o3 (83.9%) at lower cost, so batching probably does not explain the gap. Physicians spent a mean of 11.9 minutes per case closed book and 18.1 open book, and completed about 35 of the 56 CPC cases each.

Model improvements were tested with one sided paired permutation tests (10,000 resamples). The headline accuracies in the main text carry no confidence intervals, and physician accuracy is averaged over physicians who completed different numbers of cases, with no analysis that accounts for clustering by physician and case. The main text does not say whether the 83.2% and 80.3% figures are computed on all cases or only on the held out sets the physicians saw, so the headline comparison may not be on identical cases. On CPC cases the orchestration gain was significant for six of the fifteen models tested (Claude 4.5 Sonnet, Gemini 2.5 Flash, GPT‑5, GPT‑OSS 120B, Llama 4 Maverick, DeepSeek R1).

Two behavioral findings. DxO asked a mean of 25.5 questions before ordering its first test on CPC cases; physicians asked 5.6. And when DxO was wrong, it usually named a neighbor of the right answer (the infection without its source, the cancer without its subtype), while wrong physician answers were further from the reference.

• Authorship and money. All fifteen authors are Microsoft employees holding company equity. Eight are inventors on US patent application 19/722,651 covering the evaluation framework. No funding came from outside Microsoft.
• Commercial context. Microsoft's June 2025 announcement cited more than 50 million health sessions a day across Bing and Copilot. Dominic King, a coauthor and Microsoft's health lead, told Gizmodo that testing will begin at Beth Israel Deaconess within months.
• Overlapping authorship. Eric Horvitz, senior author here, is also an author on the three comparison trials the paper cites (Goh 2024, Brodeur 2026, Everett 2026). Adam Rodman of Beth Israel Deaconess, Robert Wachter and Eric Topol are thanked in the acknowledgments for their insights. That is not a financial conflict, and Topol's post about the paper does not mention it.
• Changes since the preprint. Peer review removed "superintelligence," added the open book and ED arms, and produced a limitations section that concedes the specialist, real practice, cost estimate and safety limits. It does not address the cost incentive for questioning, the model family coupling or the ED incorporation problem. The paper says plainly that it does not establish superiority over specialists or real practice.
• Reproducibility. The SDBench code is released under MIT license. The CPC data require an NEJM license; the ED subset is promised on PhysioNet under credentialing. Independent replication is possible in principle.

Across these studies, models working alone have usually scored above unaided physicians on the study's own metric, most clearly on curated cases that reward breadth of recall. Putting the model in the physician's hands produces smaller gains, or none. Brodeur is the closest thing to a real world check, because it used actual ED charts at the moment of triage rather than a simulated patient, and the margin there was about 11 to 18 points on a different accuracy measure. SDBench's contribution is the sequential format and the cost axis, which none of the others attempt.

Eric Topol posted the paper on 22 October with the summary "Higher medical accuracy at lower cost," describing a three agent system that nearly doubled physician accuracy and also beat physicians using AI. In follow ups he noted it made the Science cover, said it was the first demonstration he knew of AI cutting testing and cost while improving accuracy, and confirmed physicians worked open book. The post drew about 436 likes and 34 replies.

• Against the paper. Nearly double understates CPC (2.3 times) and overstates ED (1.7 times). There is no physician plus AI arm. The benchmark has three agents (Gatekeeper, Diagnostic, Judge); the orchestrator has five roles. Lower cost holds only for the budgeted configurations. The post omits the closed book arm and does not mention that Topol is thanked in the paper.
• Roger Seheult (MedCram) asked whether the study measured recovery from an early wrong diagnosis and any harm from it. It did not; the Judge scores only the final answer, and the paper says therapeutic appropriateness and safety are out of scope.
• Ryan Shea (former FDA senior advisor) argued the comparator models are over a year old. Partly fair: the baselines are 2025 models (GPT‑5, Claude 4.1 Opus, Claude 4.5 Sonnet, Gemini 2.5 Pro). The supplement's GPT‑5.5 rerun answers some of this; newer Anthropic and Google models were not run.
• Other replies raised the zebra problem (CPCs are the hardest teaching cases) and asked whether physician plus AI was tested. Steven Quay quote posted it as a big deal for medical AI. None of the replies retrieved raised the conflict of interest, though most of the 34 were not retrieved.