What Adam Is Reading
Tell It Where It Is
A Stanford and MIT group automated the search over the scaffolding around a frozen model, and found a ten point spread on the same weights. The scaffolding is the part nobody in health care is buying deliberately.
Single-paper review · 4 sources · August 2026

When you buy a clinical AI tool you think you are buying a model. You are buying a model wrapped in perhaps two thousand lines of code that decides what the model sees, in what order, how many tries it gets, and what happens when it fails. That wrapper is called the harness. Nobody puts it in the contract.

A group from Stanford, MIT and KRAFTON published a preprint in March asking what happens if you stop hand tuning the harness and search over it automatically. They call the result Meta-Harness. The interesting part is not the method. It is the size of the gap it reveals.

Holding the model completely fixed at Claude Haiku 4.5, six different harnesses on TerminalBench-2 produced a ten point spread. Claude Code scored 27.5 percent. Meta-Harness scored 37.6 percent. Same weights, same tasks, same everything except the code around the model.

What they found
1
Raw execution traces beat model written failure summaries
The ablation

This is the finding I would keep. Feeding the search raw execution traces produced a median of 50.0. Feeding it summaries that a model had written about why the previous attempt failed produced 34.9, which is no better than feeding it the bare scores.

Asking a model to explain its own failure and then acting on the explanation is worse than showing it what actually happened. The summary is where the information goes to die.

Solid
2
The method is expensive in a way the headline hides
The number

Ten million tokens of feedback context per iteration, against 0.026 million for the comparison method. That is a factor of roughly four hundred. It is a real result and it is not a cheap one, and the cost is the thing that determines whether anyone outside a lab can run this.

Solid
3
Three things in the framing that will not survive being repeated
What to watch

The six times figure in the paper's opening sentence is a citation to prior work. It is already being reattributed to this paper by people writing about it, which is how a number becomes folklore.

The abstract claims 4.7 points across five held out models. The search model is included in that average. Four are genuinely held out.

Search and evaluation both run on the same 89 TerminalBench tasks with no held out split. And the reported number two leaderboard position has been overtaken; the board now sits near 84.7 percent at the top across 142 entries. Benchmark rankings in this field have the shelf life of yogurt.

Mostly Solid
Why this belongs in a health care newsletter. Procurement asks which model a vendor uses. It asks about training data, and validation cohorts, and FDA status. It does not ask what the retry policy is, how the context window gets filled, what happens on a tool error, or whether the same scaffolding runs in your instance as in the demo. A ten point swing on identical weights says that question is not a detail. Two vendors can both truthfully say they run the same model and deliver measurably different systems.
So What

The harness is the intervention. We keep evaluating the model and buying the wrapper, and the wrapper is where the ten points live.

Confidence: moderate. This is a preprint that has not been through review, the benchmark it uses has no held out split, and the leaderboard position it cites has already moved. The ablation result on raw traces versus summaries is the part I would expect to replicate. Numbers verified against the preprint as posted; the version of record may differ.

Sources

Primary: Lee, Nair, Zhang, Lee, Khattab and Finn. "Meta-Harness." arXiv:2603.28052v1, March 30, 2026.

Supporting: Project page and GitHub artifact repository accompanying the preprint.

Leaderboard: Terminal-Bench 2.0 public leaderboard, checked August 19, 2026.