What Adam Is Reading
What the Token Meter Counts
A Carnegie Mellon and JetBrains preprint links five weeks of coding agent spend to the commits of 404 engineers. Checked claim by claim.
Single source review· Preprint, not peer reviewed· 9 claims· November 2026

Analysis limits. I read the full preprint (arXiv 2611.13396, posted November 10, 2026), including both supplements. I did not run the replication package or see the underlying data, which JetBrains shares only under a nondisclosure agreement. The grey literature the paper cites (the Meta leaderboard, the Amazon memo, vendor ROI reports) was not checked independently. Citation traction: not checked; the paper is three days old. The verdicts were cross checked by two other models (Grok 4.7 and GPT‑5.5) with authorship stripped; neither could read the paper, and the critiques that held up are reflected below. Claude drafted this review. Claude is made by Anthropic, which sells the tokens this paper measures and makes a coding agent of the kind it studies. No patient material appears anywhere in this piece.

In the spring, an engineer at Meta built a dashboard so colleagues could compete to be the company's top consumer of AI tokens. Mark Zuckerberg did not make the top 250. The dashboard came down after the press found it. Elsewhere, executives said a good engineer should burn tokens worth a large share of their salary, and at least one company set a monthly spending floor.

All of that assumes the meter measures work. This preprint is the first I have seen that tests the assumption on the same engineers week after week. JetBrains meters every coding agent call through an internal gateway, so the authors could line up each engineer's weekly token count against the commits that engineer made that week. The answer is a loose fit. Spend says a lot about who the engineer is and very little about what the engineer shipped that week.

The paper is careful, and it says plainly what it cannot show. Its weakness is built into the question. It tests one proxy (tokens) against another (commits), and commits have most of the flaws the title assigns to lines of code.


Who wrote it, and which way they lean

Seven authors. Shyam Agarwal and Bogdan Vasilescu are at Carnegie Mellon, where Vasilescu runs a group that studies software repositories and, lately, what AI coding tools do to them. Thomas Zimmermann is at UC Irvine and is a coauthor of SPACE, the developer productivity framework the paper discusses. Four authors work at JetBrains or JetBrains Research. The paper does not state its funding.

JetBrains sells AI coding tools, so the people who built the meter also sell what it meters. The finding does not obviously flatter the company: telling customers that usage is a poor proxy for productivity is not how you sell more usage. The authors disclose that some metered use may be JetBrains engineers testing their own agents, and that they used AI for data analysis and writing.

One dependency matters later. The mandate study the paper leans on for comparison (He et al., where an enterprise doubled throughput while raising spend at least twentyfold) shares two authors with this one. The two papers agree, and they are not independent.


The question

What the study asked. Among 404 engineers at one company, over five weeks, how closely does an engineer's weekly token spend track that engineer's weekly commits, comparing each engineer's weeks with one another and engineers with each other? And where does the spend that does not track commits go?

What the title asks. Is token spend the AI era's lines of code, a number that is easy to count and loosely related to what engineers accomplish?

The gap. Accomplishment is never measured. Output here means commits, which miss code review, design, quality, and anything that never touches a repository. The study shows convincingly that tokens do not track commits. Whether they track value is a question the data cannot reach, and the authors mostly say so.

The study as designed
ElementAgarwal et al., arXiv 2611.13396
DesignObservational panel. Gateway telemetry joined to repository commits, plus an anonymous company survey that cannot be linked to individuals.
SettingJetBrains, about 2,600 employees. Agent use was voluntary, with no target.
Population404 metered engineers, 1,827 person weeks (June 29 to August 2, 2026). 178 engineers with no metered use excluded.
ExposureWeekly tokens across two coding agents. Most tokens are context sent again every turn; generated text is under 1%.
OutcomeWeekly commits, dated by author date. Pull requests were not available.
AnalysisPoisson quasi maximum likelihood with person and week fixed effects, cluster bootstrap intervals, held out cross validation.
Survey412 respondents (15.8% of employees); 167 technical evaluations of the two agents answered the usage questions.
DisruptionA new frontier model arrived in week 3 and took 45% of all spend within two weeks.

Claim by claim
1
Spend is mostly a property of the engineer
What the paper says

Token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over five weeks.

What the evidence shows

Knowing who the engineer was explained 79% of the variation in weekly tokens. Split by odd and even weeks, rank agreement was 0.77, projected to 0.87 for the five week total. Role, manager status, and team predicted held out engineers worse than the weekly average, so the person effect is not simply job title in disguise.

What to hold loosely

Five weeks is a short window for the word trait. The 79% is an in sample figure, and the person effect could still reflect a project or a habit of long sessions that changes next quarter. Single weeks rank engineers far less consistently (0.4 to 0.6).

Mostly Solid
2
Twice the spend, about 10% more commits
What the paper says

Within an engineer, doubling spend was associated with about 10% more commits.

What the evidence shows

The slope of commits on tokens is 0.13 (95% interval 0.03 to 0.32), which works out to about 9% more commits for twice the spend, with a plausible range of 2% to 25%. In the other direction, a week with twice the usual commits had about 20% more tokens (elasticity 0.26). Correcting for noise in commit counts by using lines changed raises that to 0.48, still far from the 1.0 that proportional spending would require. Spend and commits peak on the same day, so the weak link is not output arriving late.

The caveat

This is an association. Engineers chose how much to use the agents, and the authors say so in the abstract.

Solid
3
At similar output, spend differs 39 to 240 fold
What the paper says

Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th.

What the evidence shows

The figure is accurate for the five commit quintiles. The 240 comes from the lowest quintile (0 to 2 commits).

What the headline stretches

A ratio of percentiles grows without limit as the 10th percentile approaches zero, and many weeks belong to light users who barely touched the agent. The steadier number is per engineer: over five weeks, the 90th percentile engineer used 71 times the tokens of the 10th. That is still enormous, and it makes the same point without the drama.

Mostly Solid
4
Up to two fifths of spend is not tied to the week's output
What the paper says

Up to about two fifths of spend does not vary with the week's commits.

What the evidence shows

The authors give honest bounds. Weeks with no commit at all hold 3% to 4% of tokens. A curve fitted to each engineer's spend at zero output puts the upper bound at 39% to 42%; a simpler curve that gives 89% fails every fit diagnostic and is rightly set aside. The upper bound includes the fixed cost of using an agent at all and the spend behind commits written by hand.

What the phrase invites

"Not tied to output" will be read as waste. The paper does not claim that, and its own survey suggests much of it is analysis, research, and planning that never becomes a commit. Read the 39% as spend the commit count cannot explain. The paper's own estimates range from 3% to 42%.

Mostly Solid
5
Nearly nine in ten used the agent beyond coding
What the paper says

In nearly nine of ten agent evaluations, technical respondents reported using agents for tasks beyond coding.

What the evidence shows

89% of the 167 evaluations that answered the usage questions named at least one noncoding activity: analyzing data (52%), brainstorming (49%), research (44%), planning (39%). Among the 100 evaluations of tools the survey assigned rather than ones respondents chose, the figure is 83%.

What to hold loosely

16% of employees answered, by choice, at a company that builds these tools. The direction is believable. The precise share is a property of who answered.

Mostly Solid
6
Spend measures use, "not the outcome and value of that work"
What the paper says

Token spend measures how much engineers use agents across many kinds of work, and not the outcome and value of that work.

What the evidence shows

The first half is well supported: spend tracks the engineer, the session length, and the model in use far better than it tracks commits.

The stretch

Value was never measured. The paper shows spend is a poor proxy for commits, and commits are themselves a poor proxy for value. "Not the value" is a reasonable inference and an untested one. In the discussion the authors are more careful than in the abstract.

Mostly Solid
7
A median of ten hours saved a week
What the paper says

In 91% of evaluations respondents said the agent saved them time that week, with a median of 10 hours (interquartile range 5 to 20).

What the evidence shows

The numbers are reported accurately, and the paper flags them as self report, citing the METR trial in which experienced developers took 19% longer with AI tools while believing they were faster.

What the reader should weigh

The hours figure leaves out costs the same survey found. 55% of respondents handled colleagues' material they suspected was AI generated, at a mean of 1.6 hours a week, and team leads bore more of it. 31% reported higher cognitive load. The report is accurate. The evidence behind it is the weakest in the paper.

Solid (as self report)
8
Spending targets won't raise output much
What the paper says

Companies should not expect spending targets to raise output much. If the within engineer association were causal, doubling spend through a target would raise commits by about 10%.

What the evidence shows

The authors state the condition. Their data cannot answer it, because nobody was assigned a budget. The closest outside evidence is the He et al. mandate study, where throughput doubled while spend rose at least twentyfold. That implies a similar weak elasticity, though a "twentyfold or more" makes the comparison approximate, and the study shares two authors with this one.

Why it stays open

A target changes behavior: who uses the agent, on what, and how much padding creeps in. An observational slope cannot predict that. The conclusion is plausible and untested here.

Open Question
9
Tokens per commit can't pick your most wasteful engineers
What the paper says

Dividing spend by commits does not show how efficiently engineers use agents.

What the evidence shows

Four reasons, all from the data. The ratio falls automatically in busy weeks, because spend rises more slowly than commits. It ranks engineers consistently overall (0.80) but not at the top: of the 34 engineers in the top decile by tokens per commit in odd weeks, only 17 remained there in even weeks. It treats agent work that ends without a commit as pure cost. And a model release moved it: weeks on the new model used about 45% more tokens at the same commit count.

Solid
What the paper leaves out
  • Quality. No measure of whether heavier spend produced better or worse code. Next week's bug fixes did not rise after heavy weeks, which is reassuring and thin.
  • What the gateway doesn't see. Chatbots and other assistants are not metered, so "light users" may simply use AI elsewhere. The 178 unmetered engineers committed slightly more in the period before the study.
  • The larger cost. Engineers' own time is not in the data. Tokens are the smaller bill.
  • The setting. One company, mature codebases, five summer weeks, a model change in the middle, and engineers who build developer tools for a living.

Logical gaps
IssueWhere it appears
Proxy tested against proxyThe title asks about productivity; the data compare tokens with commits. The paper is right that the two diverge. It cannot say which is closer to value.
Unexplained read as unproductive"Not tied to output" in the abstract. The body says this spend may be useful work the repository never records. Readers will take the abstract.
Model choice among curve shapesFour shapes give 39% to 89%. The authors show all four and choose by stated diagnostics, which is the honest way to handle it.
AI coding of free textSurvey themes were coded by an AI tool and checked by one author. No second human coder or agreement statistic.
Agreement that isn't independentThe external study that matches this one shares two authors and the same research program.

Lines of code, again

Lines of code survived decades of criticism because program size really did track effort well enough to price a project. Tokens don't even have that excuse. Lines count something an engineer produced. Tokens count something an engineer consumed, recorded whether or not anything came of it, and inflated each time a vendor ships a model that reads more context per turn.

The more useful finding is about when the meter is fine. Summed over five weeks, spend ranks engineers consistently, so last month's bill is a fair guide to next month's budget. The authors' advice follows from that. Use tokens for accounting. Cut cost by caching and trimming context before capping people, because the heaviest users also produce most of the output. To judge value, record which task each agent session served, and measure what happened to the task.

Why a hospital should care

Health systems are building the same dashboards. Ambient scribe programs report the share of encounters with an AI drafted note. Enterprise chat tools report weekly active users. Vendor ROI slides divide license cost by notes generated. Each is a usage count presented as value, and this paper shows how far apart the two can drift even when nobody is gaming the number.

The questions a governance committee should ask are the paper's own. What task did the tool serve? Did the note get better, did the clinician leave on time, did the claim go out right the first time? And who absorbed the cost downstream? The paper's 1.6 hours a week spent on colleagues' AI output has an obvious clinical cousin: the time a physician spends reading and correcting a note someone else's tool wrote.

What would settle it
  • Evidence I'd want. Session to task logging at the gateway, joined to task outcomes. A randomized budget or cap across comparable teams. Code quality measured alongside commits.
  • What to watch. Whether other companies' telemetry shows the same weak elasticity, and whether the share of spend unexplained by output falls as engineers gain experience. Five weeks was too short to tell.
So What

Token spend tells you who uses agents and how heavily. It barely moves with what they ship in a given week, and a model release can move it 45% on its own.

Use it to set budgets. Don't use it to rank people.

Confidence: High that spend tracks weekly commits only weakly at this company (careful panel design, many sensitivity checks). Moderate that this generalizes beyond one tool maker's engineers in five summer weeks. Low on whether spend tracks value, which nobody measured. Practical call for organizations: change practice on evaluation (a measure this loosely tied to output is unfit for ranking people, causal or not); keep using spend for budgets. Preprint, not peer reviewed. Observational. Output is commits. Daniel standard: no patients or clinical encounters appear in this piece.

Sources

The paper: Agarwal S, Lvova O, Sergeyuk A, Serova A, Bryksin T, Zimmermann T, Vasilescu B. The AI era's lines of code? A case study on what an engineer's token spend measures. arXiv 2611.13396 [cs.SE], November 10, 2026. arxiv.org/abs/2611.13396. Replication scripts: github.com/CMUSTRUDEL/token-study-replication

The comparison mandate study: He H, Agarwal S, Denisov-Blanch Y, Azaletskiy P, Koyejo S, Vasilescu B. AI writes faster than humans can review: a longitudinal study of an enterprise 2x mandate. arXiv 2607.01904, 2026. arxiv.org/abs/2607.01904

Perceived vs measured speed: Becker J, Rush N, Barnes E, Rein D. Measuring the impact of early 2025 AI on experienced open source developer productivity. arXiv 2507.09089, 2025. arxiv.org/abs/2507.09089

The Meta leaderboard: Munis J. Fortune, April 9, 2026, as cited by the paper (reference 4). Not independently checked.

Productivity frameworks: Forsgren N, Storey MA, Maddila C, Zimmermann T, Houck B, Butler J. The SPACE of developer productivity. ACM Queue 2021;19(1):20–48.