In April 2026, the most advanced eye-diagnosis AI at Beijing Tsinghua Changgung Hospital was being used in 3.8% of eye exams. Forty-one times out of 1,113. The month it launched, it had done better. The model had not gotten worse at reading retinas. The workflow had gotten worse at fitting into a clinic.
That number is the paper. A team at Tsinghua built an ambitious thing (an AI-native eye clinic where a stack of agents handles intake, triage, imaging, and decision support), put it into a working hospital, watched clinicians drift away from it, and wrote down why. With the ugly usage curve included. This is rarer than it should be, and it is worth more than another benchmark.
AI-TEC (the AI-Agent Augmented Tsinghua Eye Clinic) is not one model doing one task. It is a coordinated set of agents. A pre-consultation agent collects symptoms and history. A triaging agent stratifies risk and recommends next steps. A diagnosis agent fuses the multimodal imaging that eye care runs on (acuity, pressure, fundus photos, OCT). A clinician copilot drafts the differential and treatment options, and the ophthalmologist still makes the call. A patient support agent handles follow-up and education. A coordination layer they call Agent X ties the whole thing together.
Under the hood, an open large language model (Qwen3.5-35B) and a retinal foundation model (RETFound). The premise is a move from AI-assisted care (a tool the clinician has to stop and go consult) to AI-native care (the workflow itself, rebuilt around the agents so most of the assessment is already done by the time a patient reaches the doctor).
The average hides the interesting part. On raw hospital data the model scored 0.66 on healthy eyes (a coin flip with a good haircut), 0.61 on macular degeneration, and 0.73 on glaucoma. It was already excellent at diabetic retinopathy (0.98), which is the task everyone has been optimizing for a decade. After the clean relabel, healthy climbed to 0.88, AMD to 0.92, glaucoma to 0.93. The model was always capable. It had been taught with a hospital's ordinary, disagreed-upon records, and it learned exactly that.
Lesson two: clicks killed it, and clicks brought it back. Usage peaked two months in, then fell off a cliff. The authors are blunt about the cause. Each AI-related task added clicks, and the added friction quietly drove clinicians away. So they rebuilt the interface, sped up loading, cut the interaction down to three clicks, and limited how much a doctor had to type. Usage recovered from 3.8% to 23.0% in a month. The model did not change. The number of clicks did.
Lesson three: feedback has a shelf life. Complaints gathered at a review meeting a week after clinic were useless. The errors had to be caught at the moment the AI was used, in the room, with the case still open. When they fed that live feedback back into training, AUROC nudged again from 0.9383 to 0.9482. Small gain, right principle.
The back half of the Comment is the part I would tape to a wall. The authors lay out four ways that conventional AI evaluation quietly lies to you about clinical value. None of these are new to people who deploy this stuff. Seeing them written down together, by a team that just lived through all four, is the useful thing.
Silent trials run the AI in the background and never show its output to the clinician. That is a clean way to measure a model without disturbing care.
The trapA silent trial measures whether the model detects disease. It cannot measure whether the model changes care, because it is designed to change nothing. You can pass the trial and still be useless in the room. The thing everyone actually wants (better decisions) is the one thing the design forbids you from seeing.
Model performance is scored against the hospital's own diagnoses as the reference standard.
The trapOphthalmologists, subspecialists, reading centers, and expert panels disagree with each other all the time. So a reported AUROC often measures agreement with one particular observer, not correctness. When the AI and the chart diverge, the AI is not automatically wrong. Sometimes the chart is. A high score against a noisy label is a high score against noise.
The models are trained to name diseases from images.
The trapPatients do not arrive as images. They arrive with complaints, and clinicians manage symptoms and decisions, not isolated labels. An algorithm can be superb at reading a fundus photo and still answer a question nobody in the room was asking. Diagnostic accuracy and clinical usefulness are different axes, and the field keeps reporting the first as if it were the second.
Detecting the disease is treated as the win.
The trapDetection is step one of a long chain. The patient still has to accept the referral, reach a specialist, get the diagnosis confirmed, and actually receive treatment. If any downstream link fails, a perfect screen changes nothing. Accurate detection is necessary and nowhere near sufficient, and a screening program that ignores the plumbing is buying a very expensive way to find problems it cannot fix.
This is a Comment, not a trial. The prospective datasets are small (472 images, then 348), it is a single site, and the whole thing is early. Fine. The authors do not oversell the statistics, so I will not either.
The sharper tension is one they set up themselves. They argue, correctly, that the real scoreboard is patient outcomes, not benchmarks. Then they report a process metric (usage) as the headline result. By their own standard, they have not yet shown that AI-TEC helps a single patient see better. They say the outcome data are now being collected, which is the honest answer. The honest reading back is that they built the right scoreboard and it is still blank.
One more note for the file. The senior author (Tien Yin Wong) discloses that he co-founded an eye-AI company and consults for a long list of pharma. That does not make the four traps wrong. They are right. It does mean the framing (AI-native care is the future of the clinic) arrives from someone who sells the future. Read the lessons; keep a hand on your wallet during the vision statement.
For anyone deploying clinical AI, the finding that matters is not that the model hit 0.94. It is that a 0.94 model got used 3.8% of the time until someone removed two clicks.
The bottleneck in medical AI has quietly moved from the algorithm to the workflow. Most of the money is still pointed at the algorithm.
Sources
The paper: Yan, T., Zhang, D., Chen, L. et al. "Initial lessons from real-world implementation of an AI-agent eye clinic in China." Nature Medicine (2026). Published online 10 September 2026. doi.org/10.1038/s41591-026-04631-z
Numbers cited (raw vs. annotated AUROC, the December-to-May usage curve, the three-click fix, both prospective cohorts, and Table 1 on evaluation discrepancies) are drawn directly from the Comment and its figures.