Analysis limits. This is an Ideas and Opinions piece, two pages and ten references. It reports no new data, so there is no risk of bias table. What can be checked is whether each cited source says what the essay says it says, whether the disclosures match the public record, and what relevant work the essay does not engage. Full text read from the published pages. Disclosure forms read from the Annals supplement. Findings were put to two outside models (GPT‑5, Grok 4.3) with authorship removed; both rated the findings fair, and one judged the Utah point slightly unfair to a short opinion format, which is reflected below.
The essay’s test is simple. If a clinician doing nothing lets the AI output become reality (a prescription filled, a diagnosis documented), the system is autonomous, whatever the label says. By that test, the authors argue, a lot of clinical AI already is.
They name three conditions for oversight to mean something: bounded output, failure modes you can recover after the fact, and review volume a human can actually handle. Systems that lack all three should be validated and governed as the autonomous systems they are. The argument is good. Several of the supports under it are weaker than the prose suggests, and one is misstated.
| Element | Detail |
|---|---|
| Citation | Zhou CY, Willauer A, Topol EJ, Rajpurkar P. Ann Intern Med. Published online October 27, 2026. doi:10.7326/ANNALS‑26‑02753 |
| Type | Ideas and Opinions. No original data. Ten references. Manuscript went through at least two revisions (forms list R1 and R2). |
| Authors | Pulmonary and critical care, University of Cincinnati (Zhou); gastroenterology, MGH (Willauer); Scripps Research Translational Institute (Topol); Department of Biomedical Informatics, Harvard Medical School (Rajpurkar, corresponding) |
| Contributions | Conception and design by Zhou and Rajpurkar. Drafting by Zhou and Willauer. Data collection by Zhou. |
| Core claim | Human in the loop persists nominally while dissolving operationally. What remains distributes liability, mostly onto the physician. |
| Proposal | Validate by operational behavior instead of nominal classification. Certification proportionate to capability, independent adversarial testing before deployment, postdeployment surveillance against outcome thresholds instead of physician concurrence rates. |
| Funding | None reported for the manuscript. |
| The essay says | The source shows | Verdict |
|---|---|---|
| Utah’s medical board called for suspension of the Doctronic renewal pilot in April 2026; the state declined, citing physician review of every refill (ref 1) | Correct. The Office of AI Policy said phase one “requires a licensed medical practitioner’s approval before issuing a renewal.” Phase two begins after 250 renewals in each of nine drug classes and moves to 10% sampling. | Solid |
| Overreading cardiologists corrected only 72% of computer misread AF, versus 94% reading primarily (ref 4) | Numbers correct. But both groups saw the same wrong computer label. The 94% group was the ordering cardiologist who knew the patient; the source’s authors attribute the gap to clinical context. The comparison is context against no context. Both groups were aided. The 28% that overreaders missed still points to anchoring. | Mostly Solid |
| Clinicians unbiased when unaided became biased with biased recommendations (ref 5) | Correct, in a web experiment (438 clinicians) on calling police or medical help in a mental health crisis. Not an LLM. The paper’s headline result is that framing advice as descriptive flags instead of prescriptive recommendations kept decisions unbiased. The essay leaves that out. | Mostly Solid |
| ECG algorithms cannot be adversarially manipulated by content in the input | Uncited. Han et al. (Nature Medicine 2020) built visually undetectable adversarial ECGs that fooled a deep learning arrhythmia classifier. Prompt injection into an LLM is easier to pull off, which is the fair version of the point. | Embellished |
| A busy academic ECG service reads 160,000 a year; one insurer medical director signed off on about 60,000 denials in a month (refs 6, 3) | The Weill Cornell figure is correct (“over 160,000”). It is a whole service’s annual volume compared with one person’s monthly volume. The 60,000 figure is from ProPublica (2023, Dr. Cheryl Dopke), not from ref 3. The court order does not contain it. | Mostly Solid |
| A federal court rejected the argument that a medical director’s signature satisfies a plan’s human review requirement; denials averaged 1.2 seconds of physician time (ref 3) | The March 2025 order partly denied a motion to dismiss. It found plaintiffs had plausibly alleged a plan violation by a process that lets “an algorithm make the decision so long as a medical director pushes the button.” That is a ruling on the pleadings. No facts were found. The 1.2 seconds is an allegation in the complaint, sourced to ProPublica. Cigna called the reporting “biased and incomplete.” | Embellished |
| Trained physicians given erroneous LLM advice scored 73.3% versus 84.9%, though use was voluntary | Numbers match Qazi et al. (NEJM AI 2026): 44 physicians in Pakistan, six vignettes, ChatGPT‑4o, 20 hours of prior AI literacy training. Adjusted difference 14.0 points. Not cited in the essay. | Mostly Solid |
| A digital health company was indicted for pushing Adderall by limiting visits and information (ref 7) | Correct as of June 2024. Out of date: Done Global’s founder was convicted in November 2025 and sentenced to six years, which strengthens the point. | Solid |
| Drug interaction alerts are overridden about 90% of the time, with override rising as alert burden rises (ref 8) | Pooled override 90% (95% CI 85 to 95) across 16 studies, I² 100%. The cited meta‑analysis does not report a dose response with alert burden; it notes high override persisted even after alert counts were cut. | Mostly Solid |
| AI draft replies increased read time 21.8% without reducing reply time (ref 9) | Correct. Waitlist randomized quality improvement study, 52 UCSD primary care physicians, 2023. Reply time change −5.9%, not significant. Reply length up 17.9%. | Solid |
| Liability concentrates on the physician, the “primary locus of liability” (ref 10) | Consistent with Mello and Guha (NEJM 2024). Exact quoted wording not verified; full text paywalled. | Not Verified |
Zhou and Willauer disclose nothing. Topol discloses consulting for Abridge and Microsoft, both topically related. Abridge sells ambient documentation, which is a row in the essay’s table; Microsoft owns Nuance’s DAX ambient product. His form also answers “Yes, as disclosed above” to research grants and lists none. Rajpurkar, the corresponding author, answers no to every item, including stock and fiduciary roles.
What the public record saysRajpurkar is publicly reported as cofounder of a2z Radiology AI (founded 2024), which closed a $4.5 million seed round from Khosla Ventures and SeaX Ventures in December 2025 and holds FDA clearance for a CT triage product. The essay’s table includes imaging based diagnostic support and computer aided detection in imaging. The form asks for topically related interests over 36 months. This may have a benign explanation. It is still the kind of thing the form exists to capture.
Which way it cutsNot obviously one way. The essay’s remedy (acknowledge autonomy, certify in proportion to capability) suits patient safety. It also suits companies that want a regulatory path for products that read studies without a human reading every one. Topol and Rajpurkar have argued in Radiology for a “division of labor” in which AI handles some cases alone. Readers should know that.
The essay’s strongest sentence on utilization management rests on a motion to dismiss ruling and on figures that come from journalism, cited to the court order. The underlying point stands (plaintiffs survived dismissal on exactly this theory). The phrasing converts an allegation into a finding, and two numbers are attached to a source that does not contain one of them.
Two cited sources show that design changes oversight. Adam et al. found that descriptive flags preserved unbiased decisions. The ECG study found that clinical context, not the algorithm, explained most of the gap. The essay uses both to argue review is structurally hollow. Its own conclusion calls for better architecture, so the omission does not break the argument. It does make human review look more futile than the cited evidence shows.
The empirical spine is one 2006 retrospective chart review, one web experiment, one 44 physician vignette trial, one quality improvement study, a heterogeneous meta‑analysis, a pleading stage ruling and a news investigation. That is normal for a short opinion piece. It is thin for “meaningful per‑decision review is mathematically impossible,” which is true of one insurer’s workflow and asserted of a category.
The essay says the vocabulary for safer architectures “already exists in other domains.” It cites none of it. Bainbridge’s “Ironies of Automation” (1983) made the deskilling and vigilance argument. Elish’s “moral crumple zones” (2019) named the liability argument. Green (2022) reviewed 41 policies requiring human oversight of government algorithms and reached the same conclusion. In medicine this year alone, a Lancet Viewpoint (Abulibdeh et al., June 2026) and an npj Digital Medicine comment (van de Sande et al., July 2026) made closely related cases. Budzyń et al. (Lancet Gastroenterology and Hepatology 2025) is the best direct evidence for the endoscopy row of the table, and one author is a gastroenterologist. The contribution here is the clinical synthesis and the three condition test, which is useful. The idea itself is older.
Utah is the essay’s lead. The pilot has published early figures: the AI recommended renewal in 72% of cases and escalated 28%, physicians agreed with its renewal decisions 91% of the time, about a third of escalations were judged overcautious, and no serious safety incidents have been reported. These do not settle anything and a two page essay need not include them. They do show the pilot is still in its fully reviewed phase, which is the phase the essay’s critique does not reach. Utah has since approved five more health AI agreements (October 2026), including one that issues initial acne prescriptions.
- The functional test of autonomy (inaction produces the output) is clean and usable by a governance committee tomorrow.
- The three conditions (bounded output, recoverable failure, proportionate volume) are a reasonable screen for deciding when a signature means review.
- Monitoring patient outcomes against prespecified thresholds, instead of physician concurrence rates, is the right correction. Concurrence measures agreement, and agreement is the thing automation bias inflates.
- Override rates near 90%, longer read times with AI drafts, and accuracy loss under erroneous advice are each documented, even if each is one study deep.
- How often nominal review fails in deployed clinical systems, as opposed to vignettes, experiments and one insurer.
- Whether interface design (flags instead of recommendations, forced independent reads, sampling designs) can restore meaningful review at real volumes.
- What “certification proportionate to operational capability” would require in practice, or who would issue it.
Medical ethics and privacy check. Clean. No patient, family member or clinical encounter appears in the essay or this appraisal. The named physician in the Cigna reporting is a public figure in published journalism and is named here only to correct a citation. No permission note required.
The thesis holds and is older than the essay. Its evidence is thinner than its prose, and its best sentence about the courts overstates what a court decided. The corresponding author cofounded an imaging AI company and checked no on every disclosure box. An essay about signatures that certify nothing deserves a disclosure form that certifies something.
Sources
The essay: Zhou CY, Willauer A, Topol EJ, Rajpurkar P. The Illusion of Oversight in Clinical Artificial Intelligence: Human Out of the Loop? Ann Intern Med. October 27, 2026. doi:10.7326/ANNALS‑26‑02753
Disclosures: Author ICMJE disclosure forms, Annals supplement. acpjournals.org
a2z Radiology AI: “a2z Radiology AI: $4.5 Million Seed Funding Closed Following FDA Clearance.” Pulse 2.0, December 2025. pulse2.com. Also Radiology Business, radiologybusiness.com
Division of labor: Topol EJ, Rajpurkar P, commentary in Radiology, as reported by Radiology Business. radiologybusiness.com
Utah pilot: Utah News Dispatch, April 28, 2026. utahnewsdispatch.com. Telehealth.org, April 27, 2026 (phase thresholds). telehealth.org. Fierce Healthcare (phase design, early data). fiercehealthcare.com. Becker’s, October 2026 (expansion). beckershospitalreview.com
Cigna case: Kisting‑Leung v. Cigna, Order granting in part and denying in part motion to dismiss, E.D. Cal., March 31, 2025. Georgetown litigation tracker. ProPublica, “How Cigna Saves Millions by Having Its Doctors Reject Claims Without Reading Them,” March 25, 2023. propublica.org
ECG overreading: Anh D, Krishnan S, Bogun F. J Electrocardiol. 2006;39:343‑345. doi. Weill Cornell Coleman Center for Electrocardiography. weillcornell.org
Adversarial ECG: Han X, et al. Deep learning models for electrocardiograms are susceptible to adversarial attack. Nat Med. 2020;26:360‑363. doi
Biased advice: Adam H, et al. Commun Med. 2022;2:149. doi
Automation bias trial: Qazi IA, et al. NEJM AI. 2026;3(5):AIoa2501001. doi
Done Global: U.S. Department of Justice, conviction announcement, November 2025. justice.gov. Sentencing, Fierce Healthcare. fiercehealthcare.com
Alert overrides: Felisberto M, et al. Health Informatics J. 2024;30:14604582241263242. doi
AI draft replies: Tai‑Seale M, et al. JAMA Netw Open. 2024;7:e246565. doi
Liability: Mello MM, Guha N. N Engl J Med. 2024;390:271‑278. doi
Deskilling: Budzyń K, et al. Endoscopist deskilling risk after exposure to AI in colonoscopy. Lancet Gastroenterol Hepatol. 2025;10:896‑903. doi
Related 2026 arguments: Abulibdeh R, et al. Who’s really in the loop? Lancet. 2026;407:2340‑2344 (summary via HIFA, hifa.org). van de Sande D, Economou‑Zavlanos N, van Genderen ME. Meaningful oversight of medical AI beyond human in the loop. npj Digit Med, July 2026 (Duke summary).
Older foundations: Green B. The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review. 2022. arXiv:2109.05067. Elish MC. Moral crumple zones. Engaging Science, Technology, and Society. 2019. estsjournal.org. Bainbridge L. Ironies of automation. Automatica. 1983;19:775‑779.