Good clinical AI tools are backstopped by human judgment. A German team tested that safety net with forty-eight clinicians (more than a third of them final-year medical students). The clinicians listened to known recorded heart sounds in blocks of ten, each case arriving with an AI diagnosis. Some clinicians were asked to render a diagnosis before seeing the AI diagnosis. Some saw the AI diagnosis and then decided whether they agreed. The researchers deliberately made the AI worse with each block, from 90 percent accurate down to zero. The last block was 100% incorrect every time. Only one in eight clinicians noticed anything was wrong. The group that answered before seeing the AI did worse at the end, changing their answers that had been right 78 percent of the time and finishing at 52 percent.
The AI’s answer came with implied veracity. In other studies, experience softens the AI-induced bias. Here senior physicians and students both changed their answers. The lesson is that thinking alongside AI is a skill unto itself. We need to build AI tools that prompt this critical thinking and start teaching students to work in our new world of augmented intelligence.
In the last ten cases of this experiment, the AI was wrong every time. The heart sound in the headphones and the tracing on the screen came from two different children. Participants had been told the system was 96 percent accurate.
Three in each group of 24 noticed anything had changed.
The paper, from Nadine Schlicker and colleagues at Marburg, Freiburg and Aalborg, tests an assumption written into most clinical AI policies, and into the EU AI Act's human oversight article, which names automation bias in its text. If the machine breaks, the doctor will catch it. The answer here is mostly no, and a popular fix did not help.
Start with the part that confuses people. There was no AI that slowly went bad. The "AI" was a label (normal or abnormal) that the researchers wrote in advance for every case. They decided exactly which answers it would get wrong, and they made it worse in steps. A real system can drift after a software update, a new stethoscope, a shift in the patient mix or an attack. Nobody can ethically wait for that to happen on real patients, so the researchers staged the failure and watched who noticed.
Participants heard 40 recorded heart sounds, with a phonocardiogram and spectrogram on screen, and called each one normal or abnormal. The recordings came from CirCor, a public Brazilian pediatric dataset, so every patient was a newborn, toddler, child or adolescent. Each block held five normal hearts and five with murmurs, and the AI's mistakes leaned toward false alarms, the way real screening tools do.
1 false alarm
3 false alarms, 1 miss
a coin flip
5 false alarms, 5 misses
Everyone could replay each recording as often as they liked, and everyone rated "I trust the system" after every fifth case. The outlined box is the useful trick. Because the forcing arm committed before seeing the AI, their first answers measure how well these clinicians do alone, case by case, with the same recordings. That baseline is what makes the results readable. Error counts are from the authors' public trial file; it shows 2 false alarms and 3 misses in block 3, where the methods text says 3 and 2.
Figure A drawn for WAiR from the paper's methods and the authors' GitHub stimulus files.
Why a staircase rather than a cliff? The middle steps ask whether anyone notices a system getting worse before it fails outright. The last step is the cliff with a fire alarm attached. If people miss both, the problem is not subtlety.
Shaded area: the gap between what the forcing arm got right alone and what they finally answered after the AI weighed in. Block 4 final values are reported in the paper's text; the other points are approximate, read from its Figure 1.
Figure B redrawn for WAiR from Schlicker et al., Figure 1A (CC BY 4.0).
| Block | AI accuracy | Forcing group, own answer | Forcing group, final | AI first group |
|---|---|---|---|---|
| T1 | 90% | ~73% | ~82% | ~81% |
| T2 | 60% | ~76% | ~75% | ~72% |
| T3 | 50% | ~77% | ~68% | ~63% |
| T4 | 0% | ~78% | 52.1% | 64.6% |
Diagnostic accuracy by block. T4 final values are reported in the text; the rest are approximate, read from Figure 1.
The forcing group's own first answers were about 78 percent correct in the last block. After seeing a machine that was wrong on every case, their final answers were 52 percent correct. As a group, they had the right answer in hand and traded it for the machine's.
The rest follows the same shape. At 90 percent reliability the human and AI together scored about 81 percent, below the AI alone. People overrode the machine when it was right. That matches the 2024 Vaccaro meta analysis of 106 experiments, which found human and AI pairs beat humans alone but, on average, not the better of the two. From block 2 on, the humans alone were more accurate than the AI, and adding the AI made them worse. Only 12.5 percent spontaneously reported a decline in reliability. Asked directly, 37.5 percent said they had noticed.
In the forcing group, confidence in their own unassisted answers fell across the blocks while the accuracy of those answers held steady or rose. Their trust in the AI went up at block 3, to about 3.8 on a five point scale, while it was right half the time. Across both groups, mean trust moved only from 3.40 to 3.03. The machine was failing, and people doubted themselves.
Accuracy fell as the AI degraded, in both arms. Few noticed. The final block, where the AI was wrong every time and the display was visibly mismatched, is about as loud as a malfunction gets, and final accuracy was 52 to 65 percent. It fits a long human factors literature on automation complacency, and the Budzyń colonoscopy study, where adenoma detection without AI fell from 28.4 to 22.4 percent after endoscopists got used to the AI.
What limits itThis was a lab task with no patient, no history and a forced binary call at 50 percent prevalence. The failure was abrupt and total. Real drift is slower and patchier, which the authors note probably makes detection harder. GPT‑5.2 added a fair point. With ten cases a block, a careful person has little data to notice a change from 90 to 60 percent.
Answering first did not protect anyone. In block 4 the forcing group did worse than the AI first group (52.1 against 64.6 percent). The authors had predicted forcing would help and say plainly in the abstract that it did not.
What limits itThe forcing group made two judgments and two confidence ratings on every case, so by block four they had done roughly twice the decisions. Fatigue is a competing explanation that the fixed block order cannot separate. The Group by Block interaction for accuracy is modest (p = 0.037 after correction, generalized eta squared 0.05) in 24 people per arm. This tested one form of forcing in one task. It does not retire the idea.
The pattern is visible in Figure 1 and in the reported means. The authors suggest the stated 96 percent accuracy anchored people toward doubting themselves rather than the system.
What limits itThe authors call this exploratory, and it is. Trust and confidence were each a single item. The anchoring explanation is plausible and untested, since nobody was told a different accuracy figure.
The abstract says 48 physicians. The methods say 37.5 percent were final year medical students, 39.6 percent residents and 22.9 percent experienced physicians. Nearly half (45.8 percent) reported no professional experience. Cardiologists were 11 percent, and no pediatric specialty is listed, for a task built entirely on children's heart sounds. The authors report that experience did not change the results, and their sensitivity analyses controlled for it.
The gap"Clinicians and senior students" would have been accurate. The headline reads as if practicing physicians failed, and a third of the sample had not yet practiced.
The study was registered on AsPredicted in October 2023, before data collection. The registration set AI accuracy at 96, 73, 50 and 0 percent. It predicted the AI first group would do better at high accuracy, the forcing group better at 50 percent, and the two would tie at 0 percent. It also predicted higher trust in the AI first group, and listed calibrated trust, diagnostic calibration and a multi item trust scale as primary outcomes, analyzed with an AUC comparison.
What was publishedThe paper uses 90, 60, 50 and 0 percent. It summarizes the hypothesis as forcing helping overall. The trust prediction is not discussed (the forcing group trusted the AI slightly more, though the difference was not significant). The calibration outcomes and AUC analysis do not appear in the main text, and the sound and picture mismatch in the last block was not registered. None of these changes is flagged as a deviation. Some may be in the supplement, which I did not read. The core findings would likely survive. The reader should still have been told.
The methods say each block "contained the same ten trials," which Grok read as the same ten recordings repeated four times. The GitHub stimulus files show 40 distinct recordings, ten per block, with ten mismatched files in the last block. The sentence means every participant saw the same ten per block in a shuffled order. The design holds. The wording is ambiguous.
The methods describe three false positives and two false negatives in block three. The public trial file has four "abnormal" AI calls in that block, which at a 50/50 split implies two false positives and three false negatives. It changes nothing important. It does suggest the description and the files drifted apart.
Martin Hirsch, the senior clinical author and director of Marburg's Institute for AI in Medicine, cofounded Ada Health, the symptom checker company, and the university lists him as continuing in an advisory role. The paper declares no competing interests. Ada makes nothing for auscultation, and the paper's conclusion (move more of the oversight burden onto developers and hospitals) does not favor a developer. Funding was the German Research Foundation.
The week this paper appeared, KevinMD ran an essay by Harvey Castro, an emergency physician, on how to use AI without losing clinical judgment. His central rule is commit before you consult. Write your own assessment and plan, then open the AI, then account for every difference between the two. It is sensible advice. It is also, nearly to the letter, the arm of this experiment that did worse.
The two do not quite collide. Castro’s rule has a second half the German experiment lacked. Participants here committed first and could then switch to the AI’s answer silently, with no one asking why. Castro asks clinicians to reconcile every difference, out loud when teaching. The defensible reading is that committing first is not enough by itself, and that having to justify the switch may be the part that matters. Nobody has tested that.
On experience, the wider literature is kinder than this study. In the Radiology mammography experiment Castro cites (Dratsch and colleagues, 2023), 27 radiologists read 40 mammograms, a dozen of them with a wrong AI suggestion. When the AI was wrong, very experienced readers fell from 82 to 46 percent correct. Inexperienced readers fell from 80 to 20. Experience softened the pull and did not cancel it. (Castro describes all 27 readers as averaging more than 15 years of experience; that figure belongs to the most senior group only.) In this auscultation study, experience made no measurable difference, though with about 11 experienced physicians in each arm it had little room to show one.
Castro’s practical suggestions fit the authors’ conclusion better than his headline rule does. Keep some cases unassisted. Track performance with the tool switched off. Teach trainees to reason without AI before they learn to calibrate it. Those are oversight measures built into the system and the training, which is where this paper says oversight has to live.
They offer two definitions of successful oversight. Either the human and AI together beat the AI alone, or the pair at least does not fall below the human alone. This study failed the first at high reliability and the second at low reliability.
Their remedy is to stop relying on the clinician at the point of care as the main detector. Systems should show uncertainty. Deployers and developers should share accountability and monitor at scale, watching for shifts in prevalence and subgroup performance that no single clinician could see in ten cases. None of that was tested here. That part is inference.
For readers of an earlier WAiR piece, The Signature Is the Loop, this is an experimental test of the argument that essay made.
Every clinical AI policy I have read lists the physician as the safety net. This study tested the net with a machine that failed as obviously as a machine can fail, and most people fell through. The AI’s answer came with implied veracity, and experience was no defense. Thinking alongside AI is a skill of its own. We should build tools that prompt it, teach it to students, and watch the system at the level of the organization, where drift shows up in the aggregate. The participants here heard the murmur. They believed the machine over their own ears.
Sources
Paper: Schlicker N, Tepohl C, Kholmovaia A, Steffens N, van Berkel N, Mayr A, Grgic I, Hirsch MC, Langer M. “Challenges for human oversight in detecting system malfunctions in AI‑assisted medical diagnostics.” npj Digital Medicine 9:889 (2026). doi.org/10.1038/s41746-026-03410-x
Preregistration: AsPredicted #148,334, “Human‑Oversight in AI‑assisted Cardiac Auscultation,” registered October 23, 2023. aspredicted.org
Experiment code and stimuli: IKI‑Lab, human_oversight_auscultation. github.com
Recordings: Oliveira J et al. The CirCor DigiScope Phonocardiogram Dataset. PhysioNet, 2022. doi.org/10.13026/tshs-mw03
Human and AI combinations: Vaccaro M, Almaatouq A, Malone T. “When combinations of humans and AI are useful.” Nature Human Behaviour 8:2293 (2024). PMC
Commit before you consult: Castro H. “How to use AI without losing your clinical judgment.” KevinMD, November 2026. kevinmd.com
Automation bias in mammography: Dratsch T, Chen X, et al. “Automation Bias in Mammography: The Impact of Artificial Intelligence BI‑RADS Suggestions on Reader Performance.” Radiology 307:e222176 (2023). doi.org/10.1148/radiol.222176
Deskilling: Budzyń K et al. “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy.” Lancet Gastroenterology & Hepatology 10:896 (2025). doi.org
EU AI Act: Regulation (EU) 2024/1689, Article 14(4)(b). European Commission AI Act Service Desk
Author affiliation: Institute for AI in Medicine staff page, Philipps‑Universität Marburg. uni-marburg.de
Related WAiR: “The Signature Is the Loop.” wair.ajwein.com