The incorrect million tokens

Last month I made the case for an ROI currency exchange rate: the formula you work out with financing before release that transforms KPI motion into dollars. One point of first-call resolution equals this lots of dollars. One hour of engineering time recuperated equates to that lots of.

A typical objection was a variation of the very same concern. Fine, we settled on what the advantage deserves. What did it cost?

That column is empty at the majority of business. Not “approximately understood” or “we’re dealing with it.” Empty. And an empty denominator makes the numerator worthless. You can show the KPI moved and still lose the argument, since the CFO is not moneying enhancements; she is moneying enhancements that cost less than they return.

Which brings me to the most-discussed AI budget plan story of the year, and why I believe practically everybody drew the incorrect lesson from it.

What Uber in fact lacked

We owe Uber some thanks for being this transparent. They presented Claude Code in December 2025. By February, 32% of engineers were on agentic coding tools; by March, 84%. Someplace therein, the business burned through its whole 2026 AI spending plan in 4 months, a number CTO Praveen Neppalli Naga verified to The Information in April. In June, Bloomberg reported the reaction: a difficult cap of $1,500 per worker each month, per tool

Simon Willison called the cap reasonable, and he is. Offered a budget plan set before agentic coding existed, a ceiling was the right emergency situation relocation. I would have done the exact same thing.

Look at what Uber’s COO stated when asked whether the costs was working. Andrew Macdonald, on the Rapid Response podcast: “It’s extremely tough to draw the line in between among those statistics and ‘OK, now we’re really producing like 25% better customer functions.'”

That is not an expense problem. That is a measurement problem. Uber did not cap costs due to the fact that tokens are costly. It topped costs due to the fact that it might not price what the tokens were purchasing, and you can not protect a number you can not link to anything. The cap is what you grab when the currency exchange rate column is empty.

Here is the part that ought to fret you more than the budget plan overrun. Software application advancement is the best-instrumented workflow in the business. Pull demands, cycle time, release frequency, left problems. It is the one function that currently has the scoreboard I invested last month informing everybody to construct. Uber had all of it. And expense per job still was not on the board.

If it is missing out on there, it is missing out on all over.

2 levers, among them unblemished

Take a look at how the market has actually reacted to token maxing: quotas, per-seat caps, design downgrades, approval gates, control panels. Each of those controls the number of passes you make

Not one of them touches what a pass expenses

That is the entire argument. There are 2 levers, and the market has actually been tugging on among them for 6 months while dealing with the other as if it does not exist.

The intensifying no one budget plans for

In an ignorant representative loop, the complete discussion history gets re-serialized and re-injected at every action. Message history grows linearly. Billed input tokens grow quadratically.

Run a modest 20-step loop that includes 1,000 tokens of history per action. Multiply 20 by 1,000 and you get 20,000, which is the number many people bring in their heads. The real billed input is 210,000, since action 19 spends for whatever actions 1 through 18 stated.

Long time readers will acknowledge the shape. In June, I argued that AI cloud method is a physics issue, due to the fact that 50 milliseconds of cross-region latency does not cost you 50 milliseconds; it costs you 50 milliseconds times every hop in the loop. Exact same structure, various axis. Cut 800K of scrap off one call and you conserve 80% of one call. Cut it off every hop and you conserve 80% of a quadratic.

“But we have timely caching”

Every significant service provider now marks down re-sent prefix tokens. Anthropic costs cache checks out at 0.1 x basic inputwith a 1.25 x premium on the compose. OpenAI’s more recent designs arrived on the very same 0.1 x multiplier. Google’s implicit caching runs about 75% off.

Does caching eliminate the argument? Run it and see. Very same loop, checks out at 0.1 x, composes at 1.25 x:

Caching takes 79% off the uncached costs. It is the single biggest expense lever offered and every group reading this must be utilizing it. It likewise leaves you at 2.2 times the number you had in your head. Caching flattens the quadratic; it does not eliminate it totally and it includes a caution.

Caches are keyed on the specific prefix, so any modification at the front revokes whatever behind it. If your retrieval pipeline injects newly chosen portions near the top of the context each turn, you are not simply spending for tokens that did not make their location; you are breaking the cache for every single steady token that follows. Design a 25% prefix-break rate on that exact same loop and 44,000 efficient tokens end up being 90,000.

Order the timely steady to variable: system timely, tool meanings, long-lived context, then the present turn. Determine the hit rate. That is an afternoon of work, and it secures your finest expense lever.

The part that makes this more than a FinOps memo

Typically expense decrease expenses you quality. You purchase the less expensive thing, and you get the less expensive thing.

Not here. The low-relevance context you are paying to send out is the very same context deteriorating the response. This is the load-bearing claim in the piece and there is a great deal of supplier handwaving in this area, so here are 2 peer-reviewed sources, none offering retrieval facilities.

The canonical outcome is Liu et al., “Lost in the Middle” (TACL, 2024): precision follows a U-shaped curve throughout the window, strong at the edges and weak in the middle, and it drops as input grows even on designs constructed for long context.

The one that shocked me is Du et al., “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval”(Findings of EMNLP, 2025). The authors held retrieval at best and grew the input anyhow. Precision still fell, 13.9% to 85% depending upon design and job, well inside marketed limitations. It held when the filler was whitespace. It held when the unimportant tokens were masked out completely. The majority of the damage landed inside the very first 7K.

The tokens you did not require are not complimentary even when they are cached. They are watering down the thinking you are spending for. More affordable and more precise, exact same relocation. That is not a compromise; it is a mispricing.

The concern ends up being: of the million tokens I might send out, which 200K make their location?

That is not a procurement concern or a policy concern. It is a retrieval concern, addressed out of your RAG pipeline or your agentic memory. And unlike a costs cap, it has an engineering response.

6 methods to choose the best 200K

Most affordable to hardest, with business repercussion beside each, since a CIO who will never ever touch a chunking technique still requires to understand what avoiding it costs.

One, 2 and 5 are checkable today without a spending plan cycle.

The number you reclaim to fund

Number 6 should have taking out of the list, since it is the just one that produces a figure instead of an enhancement.

Expense per job. Not expense per token, which determines your supplier’s prices. Not expense per seat, which determines your headcount. Expense per solved ticket, per combined pull demand, per closed claim. The denominator under the currency exchange rate. The other 5 are how you enhance it; this one is how you report it, and it is what lets you argue the costs up when it is making. No business under a blanket cap can have that discussion.

Objections, and one sincere caution

“Context windows keep growing and designs keep improving at utilizing them.” Both real, and neither makes paying for unimportant tokens logical. The Du outcome recommends length itself brings an expense that ability gains have actually not removed, and the compounding is structural: it worsens as representatives get more self-governing, not much better.

“We currently do RAG.” A pipeline with fixed-size portions, no reranker and no eval is tokenmaxxing with additional actions.

“Caps worked for Uber.” They managed the budget plan, which was the instant issue. Ask what they did to cost per delivered function, and whether your finest engineers are now the ones allocating hardest.

The caution, due to the fact that I would rather state it than have it stated to me: some work really desire the entire file in the window. Long-form legal evaluation, whole-codebase refactors, anything where relationships in between remote areas are the point.

The journal

A costs cap manages the expense. It is the ideal emergency situation relocation and an irreversible admission that you never ever constructed the measurement.

Expense per job is the other column. It informs you the distinction in between a pricey work and an inefficient one, and those are not the very same thing.

Context length is the lever you do not manage. Expense per pass is the one you do.