What Adam Is Reading
Overruled by a Broken Machine
Forty eight clinicians listened to children's hearts with an AI that researchers broke on purpose, a little more every ten cases. One in eight noticed. The group asked to answer first, before seeing the AI, did worse at the end.
Single source review · npj Digital Medicine, Brief Communication · November 15, 2026
In brief

Good clinical AI tools are backstopped by human judgment. A German team tested that safety net with forty-eight clinicians (more than a third of them final-year medical students). The clinicians listened to known recorded heart sounds in blocks of ten, each case arriving with an AI diagnosis. Some clinicians were asked to render a diagnosis before seeing the AI diagnosis. Some saw the AI diagnosis and then decided whether they agreed. The researchers deliberately made the AI worse with each block, from 90 percent accurate down to zero. The last block was 100% incorrect every time. Only one in eight clinicians noticed anything was wrong. The group that answered before seeing the AI did worse at the end, changing their answers that had been right 78 percent of the time and finishing at 52 percent.

The AI’s answer came with implied veracity. In other studies, experience softens the AI-induced bias. Here senior physicians and students both changed their answers. The lesson is that thinking alongside AI is a skill unto itself. We need to build AI tools that prompt this critical thinking and start teaching students to work in our new world of augmented intelligence.

Analysis limits. I read the full seven page paper, the October 2023 preregistration on AsPredicted, and the public experiment code and stimulus files on GitHub, including the file that holds the AI's answer for each of the 40 cases. I also read Harvey Castro’s November 2026 KevinMD essay and checked its mammography figures against the original Radiology paper. I did not read the online supplement or reanalyze the OSF data. Numbers marked approximate were read by eye from the paper's Figure 1. Two outside models (GPT‑5.2 and Grok 4.6) checked my claims with authorship stripped. Where they disagreed with me, I say so below.

In the last ten cases of this experiment, the AI was wrong every time. The heart sound in the headphones and the tracing on the screen came from two different children. Participants had been told the system was 96 percent accurate.

Three in each group of 24 noticed anything had changed.

The paper, from Nadine Schlicker and colleagues at Marburg, Freiburg and Aalborg, tests an assumption written into most clinical AI policies, and into the EU AI Act's human oversight article, which names automation bias in its text. If the machine breaks, the doctor will catch it. The answer here is mostly no, and a popular fix did not help.


What they did

Start with the part that confuses people. There was no AI that slowly went bad. The "AI" was a label (normal or abnormal) that the researchers wrote in advance for every case. They decided exactly which answers it would get wrong, and they made it worse in steps. A real system can drift after a software update, a new stethoscope, a shift in the patient mix or an attack. Nobody can ethically wait for that to happen on real patients, so the researchers staged the failure and watched who noticed.

Participants heard 40 recorded heart sounds, with a phonocardiogram and spectrogram on screen, and called each one normal or abnormal. The recordings came from CirCor, a public Brazilian pediatric dataset, so every patient was a newborn, toddler, child or adolescent. Each block held five normal hearts and five with murmurs, and the AI's mistakes leaned toward false alarms, the way real screening tools do.

Figure A · The method
How the researchers broke the AI, ten cases at a time
96%
What every participant was told. The AI was validated at 96 percent accuracy (97 percent sensitivity, 89 percent specificity), figures borrowed from a published pediatric auscultation study. Nobody was told it would change.
90%
Block 1
Cases 1 to 10
1 wrong
1 false alarm
60%
Block 2
Cases 11 to 20
4 wrong
3 false alarms, 1 miss
50%
Block 3
Cases 21 to 30
5 wrong
a coin flip
0%
Block 4
Cases 31 to 40
10 wrong
5 false alarms, 5 misses
Plus a planted clue. The sound and the tracing came from two different children.
Time runs left to right. Every participant got the blocks in this same order, with an optional break after case 20 and no warning at any step.
AI right AI wrongBar height = how often the AI was right in that block

The two ways a single case could run
AI First (24 people)
Patient card→ AI says normal or abnormal→ Listen and look→ Final call + confidence
AI Second, the cognitive forcing arm (24 people)
Patient card→ Listen and look→ Own call + confidence→ AI says normal or abnormal→ Final call + confidence

Everyone could replay each recording as often as they liked, and everyone rated "I trust the system" after every fifth case. The outlined box is the useful trick. Because the forcing arm committed before seeing the AI, their first answers measure how well these clinicians do alone, case by case, with the same recordings. That baseline is what makes the results readable. Error counts are from the authors' public trial file; it shows 2 false alarms and 3 misses in block 3, where the methods text says 3 and 2.

Figure A drawn for WAiR from the paper's methods and the authors' GitHub stimulus files.

Why a staircase rather than a cliff? The middle steps ask whether anyone notices a system getting worse before it fails outright. The last step is the cliff with a fire alarm attached. If people miss both, the problem is not subtlety.


What they found
Figure B · The results
They had the right answer, then took the machine's
100%80%60% 40%20%0% Block 1Block 2Block 3Block 4 AI 90%AI 60%AI 50%AI 0% 78% alone forcing arm, own call 65% AI First 52% forcing arm after seeing the AI 0% the AI AI made them worse Block 1: the AI alone beat the human and AI pair

Shaded area: the gap between what the forcing arm got right alone and what they finally answered after the AI weighed in. Block 4 final values are reported in the paper's text; the other points are approximate, read from its Figure 1.

Figure B redrawn for WAiR from Schlicker et al., Figure 1A (CC BY 4.0).

BlockAI accuracyForcing group, own answerForcing group, finalAI first group
T190%~73%~82%~81%
T260%~76%~75%~72%
T350%~77%~68%~63%
T40%~78%52.1%64.6%

Diagnostic accuracy by block. T4 final values are reported in the text; the rest are approximate, read from Figure 1.

The forcing group's own first answers were about 78 percent correct in the last block. After seeing a machine that was wrong on every case, their final answers were 52 percent correct. As a group, they had the right answer in hand and traded it for the machine's.

The rest follows the same shape. At 90 percent reliability the human and AI together scored about 81 percent, below the AI alone. People overrode the machine when it was right. That matches the 2024 Vaccaro meta analysis of 106 experiments, which found human and AI pairs beat humans alone but, on average, not the better of the two. From block 2 on, the humans alone were more accurate than the AI, and adding the AI made them worse. Only 12.5 percent spontaneously reported a decline in reliability. Asked directly, 37.5 percent said they had noticed.

In the forcing group, confidence in their own unassisted answers fell across the blocks while the accuracy of those answers held steady or rose. Their trust in the AI went up at block 3, to about 3.8 on a five point scale, while it was right half the time. Across both groups, mean trust moved only from 3.40 to 3.03. The machine was failing, and people doubted themselves.


What holds up
1
Clinicians did not reliably catch a failing AI
What the data show

Accuracy fell as the AI degraded, in both arms. Few noticed. The final block, where the AI was wrong every time and the display was visibly mismatched, is about as loud as a malfunction gets, and final accuracy was 52 to 65 percent. It fits a long human factors literature on automation complacency, and the Budzyń colonoscopy study, where adenoma detection without AI fell from 28.4 to 22.4 percent after endoscopists got used to the AI.

What limits it

This was a lab task with no patient, no history and a forced binary call at 50 percent prevalence. The failure was abrupt and total. Real drift is slower and patchier, which the authors note probably makes detection harder. GPT‑5.2 added a fair point. With ten cases a block, a careful person has little data to notice a change from 90 to 60 percent.

Mostly Solid
2
Cognitive forcing did not help
What the data show

Answering first did not protect anyone. In block 4 the forcing group did worse than the AI first group (52.1 against 64.6 percent). The authors had predicted forcing would help and say plainly in the abstract that it did not.

What limits it

The forcing group made two judgments and two confidence ratings on every case, so by block four they had done roughly twice the decisions. Fatigue is a competing explanation that the fixed block order cannot separate. The Group by Block interaction for accuracy is modest (p = 0.037 after correction, generalized eta squared 0.05) in 24 people per arm. This tested one form of forcing in one task. It does not retire the idea.

Mostly Solid
3
Confidence fell faster than trust in the AI
What the data show

The pattern is visible in Figure 1 and in the reported means. The authors suggest the stated 96 percent accuracy anchored people toward doubting themselves rather than the system.

What limits it

The authors call this exploratory, and it is. Trust and confidence were each a single item. The anchoring explanation is plausible and untested, since nobody was told a different accuracy figure.

Exploratory

What doesn't hold up as well
4
"48 physicians"
What the methods say

The abstract says 48 physicians. The methods say 37.5 percent were final year medical students, 39.6 percent residents and 22.9 percent experienced physicians. Nearly half (45.8 percent) reported no professional experience. Cardiologists were 11 percent, and no pediatric specialty is listed, for a task built entirely on children's heart sounds. The authors report that experience did not change the results, and their sensitivity analyses controlled for it.

The gap

"Clinicians and senior students" would have been accurate. The headline reads as if practicing physicians failed, and a third of the sample had not yet practiced.

Embellished
5
The preregistration, and the changes nobody mentions
What was registered

The study was registered on AsPredicted in October 2023, before data collection. The registration set AI accuracy at 96, 73, 50 and 0 percent. It predicted the AI first group would do better at high accuracy, the forcing group better at 50 percent, and the two would tie at 0 percent. It also predicted higher trust in the AI first group, and listed calibrated trust, diagnostic calibration and a multi item trust scale as primary outcomes, analyzed with an AUC comparison.

What was published

The paper uses 90, 60, 50 and 0 percent. It summarizes the hypothesis as forcing helping overall. The trust prediction is not discussed (the forcing group trusted the AI slightly more, though the difference was not significant). The calibration outcomes and AUC analysis do not appear in the main text, and the sound and picture mismatch in the last block was not registered. None of these changes is flagged as a deviation. Some may be in the supplement, which I did not read. The core findings would likely survive. The reader should still have been told.

Undisclosed deviations
6
Small items
Housekeeping

The methods say each block "contained the same ten trials," which Grok read as the same ten recordings repeated four times. The GitHub stimulus files show 40 distinct recordings, ten per block, with ten mismatched files in the last block. The sentence means every participant saw the same ten per block in a shuffled order. The design holds. The wording is ambiguous.

The methods describe three false positives and two false negatives in block three. The public trial file has four "abnormal" AI calls in that block, which at a 50/50 split implies two false positives and three false negatives. It changes nothing important. It does suggest the description and the files drifted apart.

Martin Hirsch, the senior clinical author and director of Marburg's Institute for AI in Medicine, cofounded Ada Health, the symptom checker company, and the university lists him as continuing in an advisory role. The paper declares no competing interests. Ada makes nothing for auscultation, and the paper's conclusion (move more of the oversight burden onto developers and hospitals) does not favor a developer. Funding was the German Research Foundation.

Minor

"Commit before you consult," tested the same week

The week this paper appeared, KevinMD ran an essay by Harvey Castro, an emergency physician, on how to use AI without losing clinical judgment. His central rule is commit before you consult. Write your own assessment and plan, then open the AI, then account for every difference between the two. It is sensible advice. It is also, nearly to the letter, the arm of this experiment that did worse.

The two do not quite collide. Castro’s rule has a second half the German experiment lacked. Participants here committed first and could then switch to the AI’s answer silently, with no one asking why. Castro asks clinicians to reconcile every difference, out loud when teaching. The defensible reading is that committing first is not enough by itself, and that having to justify the switch may be the part that matters. Nobody has tested that.

On experience, the wider literature is kinder than this study. In the Radiology mammography experiment Castro cites (Dratsch and colleagues, 2023), 27 radiologists read 40 mammograms, a dozen of them with a wrong AI suggestion. When the AI was wrong, very experienced readers fell from 82 to 46 percent correct. Inexperienced readers fell from 80 to 20. Experience softened the pull and did not cancel it. (Castro describes all 27 readers as averaging more than 15 years of experience; that figure belongs to the most senior group only.) In this auscultation study, experience made no measurable difference, though with about 11 experienced physicians in each arm it had little room to show one.

Castro’s practical suggestions fit the authors’ conclusion better than his headline rule does. Keep some cases unassisted. Track performance with the tool switched off. Teach trainees to reason without AI before they learn to calibrate it. Those are oversight measures built into the system and the training, which is where this paper says oversight has to live.


What the authors propose

They offer two definitions of successful oversight. Either the human and AI together beat the AI alone, or the pair at least does not fall below the human alone. This study failed the first at high reliability and the second at low reliability.

Their remedy is to stop relying on the clinician at the point of care as the main detector. Systems should show uncertainty. Deployers and developers should share accountability and monitor at scale, watching for shifts in prevalence and subgroup performance that no single clinician could see in ten cases. None of that was tested here. That part is inference.

For readers of an earlier WAiR piece, The Signature Is the Loop, this is an experimental test of the argument that essay made.

So What

Every clinical AI policy I have read lists the physician as the safety net. This study tested the net with a machine that failed as obviously as a machine can fail, and most people fell through. The AI’s answer came with implied veracity, and experience was no defense. Thinking alongside AI is a skill of its own. We should build tools that prompt it, teach it to students, and watch the system at the level of the organization, where drift shows up in the aggregate. The participants here heard the murmur. They believed the machine over their own ears.

Confidence is moderate. The direction of the findings is consistent with a large literature on automation bias. The size is specific to a lab task with pediatric recordings, a mixed and partly student sample, a fixed order and an abrupt, total failure. The undisclosed preregistration changes do not reverse the findings, but they lower my trust in the framing.

Sources

Paper: Schlicker N, Tepohl C, Kholmovaia A, Steffens N, van Berkel N, Mayr A, Grgic I, Hirsch MC, Langer M. “Challenges for human oversight in detecting system malfunctions in AI‑assisted medical diagnostics.” npj Digital Medicine 9:889 (2026). doi.org/10.1038/s41746-026-03410-x

Preregistration: AsPredicted #148,334, “Human‑Oversight in AI‑assisted Cardiac Auscultation,” registered October 23, 2023. aspredicted.org

Experiment code and stimuli: IKI‑Lab, human_oversight_auscultation. github.com

Recordings: Oliveira J et al. The CirCor DigiScope Phonocardiogram Dataset. PhysioNet, 2022. doi.org/10.13026/tshs-mw03

Human and AI combinations: Vaccaro M, Almaatouq A, Malone T. “When combinations of humans and AI are useful.” Nature Human Behaviour 8:2293 (2024). PMC

Commit before you consult: Castro H. “How to use AI without losing your clinical judgment.” KevinMD, November 2026. kevinmd.com

Automation bias in mammography: Dratsch T, Chen X, et al. “Automation Bias in Mammography: The Impact of Artificial Intelligence BI‑RADS Suggestions on Reader Performance.” Radiology 307:e222176 (2023). doi.org/10.1148/radiol.222176

Deskilling: Budzyń K et al. “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy.” Lancet Gastroenterology & Hepatology 10:896 (2025). doi.org

EU AI Act: Regulation (EU) 2024/1689, Article 14(4)(b). European Commission AI Act Service Desk

Author affiliation: Institute for AI in Medicine staff page, Philipps‑Universität Marburg. uni-marburg.de

Related WAiR: “The Signature Is the Loop.” wair.ajwein.com