The wrong million tokens

Last month I made the case for an ROI exchange rate: the formula you negotiate with finance before deployment that converts KPI movement into dollars. One point of first-call resolution equals this many dollars. One hour of engineering time recovered equals that many.

A common objection was a version of the same question. Fine, we agreed on what the benefit is worth. What did it cost?

That column is empty at most companies. Not “roughly known” or “we’re working on it.” Empty. And an empty denominator makes the numerator useless. You can prove the KPI moved and still lose the argument, because the CFO is not funding improvements; she is funding improvements that cost less than they return.

Which brings me to the most-discussed AI budget story of the year, and why I think almost everyone drew the wrong lesson from it.

What Uber actually ran out of

We owe Uber some thanks for being this transparent. They rolled out Claude Code in December 2025. By February, 32% of engineers were on agentic coding tools; by March, 84%. Somewhere in there, the company burned through its entire 2026 AI budget in four months, a number CTO Praveen Neppalli Naga confirmed to The Information in April. In June, Bloomberg reported the response: a hard cap of $1,500 per employee per month, per tool.

Simon Willison called the cap rational, and he is right. Given a budget set before agentic coding existed, a ceiling was the correct emergency move. I would have done the same thing.

But look at what Uber’s COO said when asked whether the spending was working. Andrew Macdonald, on the Rapid Response podcast: “It’s very hard to draw a line between one of those stats and ‘OK, now we’re actually producing like 25% more useful consumer features.’”

That is not a cost complaint. That is a measurement complaint. Uber did not cap spending because tokens are expensive. It capped spending because it could not price what the tokens were buying, and you cannot defend a number you cannot connect to anything. The cap is what you reach for when the exchange rate column is empty.

Here is the part that should worry you more than the budget overrun. Software development is the best-instrumented workflow in the enterprise. Pull requests, cycle time, deploy frequency, escaped defects. It is the one function that already has the scoreboard I spent last month telling everyone to build. Uber had all of it. And cost per task still was not on the board.

If it is missing there, it is missing everywhere.

Two levers, one of them untouched

Look at how the industry has responded to token maxing: quotas, per-seat caps, model downgrades, approval gates, dashboards. Every one of those controls how many passes you make.

Not one of them touches what a pass costs.

That is the whole argument. There are two levers, and the industry has been yanking on one of them for six months while treating the other as if it does not exist.

The compounding nobody budgets for

In a naive agent loop, the full conversation history gets re-serialized and re-injected at every step. Message history grows linearly. Billed input tokens grow quadratically.

Run a modest 20-step loop that adds 1,000 tokens of history per step. Multiply 20 by 1,000 and you get 20,000, which is the number most people carry in their heads. The actual billed input is 210,000, because step 19 pays for everything steps 1 through 18 said.

Longtime readers will recognize the shape. In June, I argued that AI cloud strategy is a physics problem, because 50 milliseconds of cross-region latency does not cost you 50 milliseconds; it costs you 50 milliseconds times every hop in the loop. Same structure, different axis. Trim 800K of junk off one call and you save 80% of one call. Trim it off every hop and you save 80% of a quadratic.

“But we have prompt caching”

Every major provider now discounts re-sent prefix tokens. Anthropic bills cache reads at 0.1x standard input, with a 1.25x premium on the write. OpenAI’s newer models landed on the same 0.1x multiplier. Google’s implicit caching runs about 75% off.

So, does caching kill the argument? Run it and see. Same loop, reads at 0.1x, writes at 1.25x:

	Effective input tokens
Naive per-step estimate	20,000
Actual, uncached	210,000
Actual, fully cached	44,000

Caching takes 79% off the uncached bill. It is the single largest cost lever available and every team reading this should be using it. It also leaves you at 2.2 times the number you had in your head. Caching flattens the quadratic; it does not remove it entirely and it comes with a caveat.

Caches are keyed on the exact prefix, so any change at the front invalidates everything behind it. If your retrieval pipeline injects freshly selected chunks near the top of the context each turn, you are not just paying for tokens that did not earn their place; you are breaking the cache for every stable token that follows. Model a 25% prefix-break rate on that same loop and 44,000 effective tokens become 90,000.

Order the prompt stable to variable: system prompt, tool definitions, long-lived context, then the current turn. Then measure the hit rate. That is an afternoon of work, and it protects your best cost lever.

The part that makes this more than a FinOps memo

Usually cost reduction costs you quality. You buy the cheaper thing, and you get the cheaper thing.

Not here. The low-relevance context you are paying to send is the same context degrading the answer. This is the load-bearing claim in the piece and there is a lot of vendor handwaving in this space, so here are two peer-reviewed sources, none of them selling retrieval infrastructure.

The canonical result is Liu et al., “Lost in the Middle” (TACL, 2024): accuracy follows a U-shaped curve across the window, strong at the edges and weak in the middle, and it drops as input grows even on models built for long context.

The one that surprised me is Du et al., “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval”(Findings of EMNLP, 2025). The authors held retrieval at perfect and grew the input anyway. Accuracy still fell, 13.9% to 85% depending on model and task, well inside advertised limits. It held when the filler was whitespace. It held when the irrelevant tokens were masked out entirely. Most of the damage landed inside the first 7K.

The tokens you did not need are not free even when they are cached. They are diluting the reasoning you are paying for. Cheaper and more accurate, same move. That is not a trade-off; it is a mispricing.

The question becomes: of the million tokens I could send, which 200K earn their place?

That is not a procurement question or a policy question. It is a retrieval question, answered out of your RAG pipeline or your agentic memory. And unlike a spending cap, it has an engineering answer.

Six ways to pick the right 200K

Cheapest to hardest, with the business consequence next to each, because a CIO who will never touch a chunking strategy still needs to know what skipping it costs.

1. Filter before you search. Metadata and scope narrowing costs nothing and kills most irrelevant candidates before semantic search runs. You are paying a similarity search to rediscover what you already knew.
2. Rerank, don’t just retrieve. Vector search optimizes recall; at the point of injection, you need precision. Without it, precision is whatever your embedding model happened to give you.
3. Chunk on meaning, not character count. Fixed-size chunking splits reasoning that needed to stay together. You pay 3x for one idea and the model sees it in pieces.
4. Compact, don’t accumulate. Summarize prior turns instead of re-sending raw transcript. This is the only item that changes the shape of the curve.
5. Dedupe across turns. Agent loops re-retrieve the same chunks repeatedly and almost nobody measures it. You are paying several times per session for identical text.
6. Know when to stop. At what point does the next chunk stop paying for itself, in dollars and in dilution? No answer means you do not have a retrieval strategy; you have a default.

One, two and five are checkable this week without a budget cycle.

The number you take back to finance

Number six deserves pulling out of the list, because it is the only one that produces a figure rather than an improvement.

Cost per task. Not cost per token, which measures your vendor’s pricing. Not cost per seat, which measures your headcount. Cost per resolved ticket, per merged pull request, per closed claim. The denominator under the exchange rate. The other five are how you improve it; this one is how you report it, and it is what lets you argue the bill up when it is earning. No company under a blanket cap can have that conversation.

Objections, and one honest caveat

“Context windows keep growing and models keep getting better at using them.” Both true, and neither makes paying for irrelevant tokens rational. The Du result suggests length itself carries a cost that capability gains have not erased, and the compounding is structural: it gets worse as agents get more autonomous, not better.

“We already do RAG.” A pipeline with fixed-size chunks, no reranker and no eval is tokenmaxxing with extra steps.

“Caps worked for Uber.” They controlled the budget, which was the immediate problem. Ask what they did to cost per shipped feature, and whether your best engineers are now the ones rationing hardest.

The caveat, because I would rather say it than have it said to me: some workloads genuinely want the whole document in the window. Long-form legal review, whole-codebase refactors, anything where relationships between distant sections are the point.

The ledger

A spending cap controls the bill. It is the right emergency move and a permanent admission that you never built the measurement.

Cost per task is the other column. It tells you the difference between an expensive workload and a wasteful one, and those are not the same thing.

Context length is the lever you do not control. Cost per pass is the one you do.