Sample cases, rubrics, and code were released June 16, 2026. Three of the four top-ranked systems (AMBOSS LiSA, OpenEvidence, Glass Health) carry an asterisk: tested after that release. The authors show that deliberate memorization of the public 30% is easy to detect (near-perfect public scores, weaker private scores) and that no tested model shows the pattern. One vendor was mid-update and asked for a retest; both results are shown. RAG vendors provided API access on request.

The contamination check catches memorization. It does not catch a vendor tuning a system prompt toward the benchmark's style, for example by making answers more exhaustive, which is what an omission-weighted metric rewards. The public and private sets correlate at r = 0.95, which helps. The cleanest comparison would be RAG scores from before the release, which the paper does not separate out.

Harm of omission is scored whenever an appropriate rubric action is not explicitly stated. The autograder was "very strict" and counted implied actions as omissions. Severe omissions carry 24 times the weight of mild ones. Most severe errors were omissions; recall lagged precision almost everywhere. The length penalty applies only beyond the longest expert reference answer, and changed rankings very little.

The benchmark measures completeness of a written plan at least as much as clinical danger. Systems built to produce long, checklist-style consults (the RAG tools) are advantaged. A concise answer that a specialist would consider adequate can register as a "severe error." The authors say as much in their limitations: these are rubric-based indicators, not event rates. The "up to 24.6% severe harm" headline should be read in that light.

An empty response scored severe harm potential in 37% of cases, worse than every model. The Discussion uses this to argue "compared with what?" and that the status quo carries harm.

Because omission is harm by definition, an empty answer fails every omission check. The reference is useful as a floor for the scale. It is not evidence about real care, where the alternative to AI is usually a specialist, a clinician's own plan, or a delay, not silence. The fair "compared with what" comparator in this paper is the physician arm, not the empty one.

The rubric options were generated by eight LLMs (OpenAI, Google, Anthropic, DeepSeek), then condensed with Claude and Gemini and reviewed by physicians. Grading is by Gemini models. The authors tested Claude and Gemini Pro as alternate reviewers (κ 0.82 and 0.80) and note OpenAI models still scored best under non-OpenAI graders.

Physician rating of every option is a strong safeguard. Still, the action vocabulary came from model outputs, so answers phrased the way models phrase things may match the rubric more easily than a human's shorthand. That cuts against physicians in the human comparison. Grader self-preference looks adequately checked.

The 1:8:24 severity weights came from a grid search optimizing ranking stability, picking the lowest weights reaching Kendall τ 0.95. Excluding action categories did not change conclusions.

Optimizing for stable rankings is defensible and transparent. It is not the same as anchoring weights to clinical consequences, and absolute F1 values depend on the choice. Rankings are the more trustworthy output; absolute percentages less so.

All cases come from Stanford eConsults. Specialists modified recommendations in 61% of cases and added missing case details. Clarifying information was added because a real eConsult includes chart review. Specialists from other institutions reviewed to limit local practice patterns. Cases are low-acuity outpatient problems.

The authors handle this carefully. The reference standard is academic specialist practice, which tends to be more comprehensive than community practice, and that again favors exhaustive answers. Results do not generalize to inpatient or acute care, as the authors state.

101 volunteers recruited through social media and listservs; 47% use AI daily or often. Two cases per arm, counterbalanced, analyzed with crossed random effects. Four of 105 excluded (two incomplete, two for time-on-task outliers); conclusions unchanged with them included. Physicians wrote free-text plans with no chart and were scored by the same strict omission grader. Compensation and time limits are not described in the main text.

The intention-to-treat gain (42.2% to 47.3%) is the most reliable human result. The comparison of "physicians vs. AI alone" is less fair than it looks: models were prompted to be complete and graded on completeness, while busy physicians typing a plan naturally write less. Low physician scores partly reflect documentation behavior, not judgment. Volunteers comfortable with AI may not represent the average generalist. With two cases per arm, each physician's estimate is noisy, though the pooled design accounts for this.

If physicians had added every recommendation GPT-5.4 surfaced to their own plan, the combined plan would have outscored both the physician and the model as used (P < 0.001).

Taking the union of two lists almost always raises recall, and recall is where the scoring is concentrated. The result is real arithmetic but a weak test of complementary judgment. It does not show what happens when physicians accept everything, including the AI's inappropriate suggestions in messier cases. The more useful finding is behavioral: a mean of 1.8 messages per case suggests physicians used the tool like a search box.

Several senior authors have Google or Google DeepMind affiliations; the first author consults for Google DeepMind and Meta. Funding includes ARPA-H's Physician-AI Collaboration Teaming program. The authors run the public leaderboard. The introduction's adoption figures cite an OpenEvidence funding press release and an AMBOSS newsroom post, both companies whose products are ranked.

Disclosure is thorough, and the results do not favor Google (Gemini models land mid-pack; OpenAI leads among generalists). The ARPA-H program's premise aligns with the paper's teaming conclusions, which is worth knowing rather than worrying about. Citing vendor press releases for adoption statistics is weak sourcing and should be replaced before publication.

Version 4, seven months after v1. The model roster reflects April to June 2026. Manual testing of consumer clinical products scored lower than API testing, with different test dates.

Any specific ranking will be stale within months, and the API version of a product may not be what clinicians use. Cite the version. Treat the leaderboard, not the PDF, as the current state, and treat the methods as the durable contribution.