Your finance lead forwards the August invoice with one question attached: why is this bigger than July when every vendor spent the summer cutting prices? Both halves of that question are true at once, which is exactly why nobody on the engineering side has a clean answer ready.
Why It Matters
The cuts were real, not marketing. Jefferies, working from Silicon Data's pricing index, tracked average inference pricing down to a 2026 low in the first week of August, crediting reductions of up to 80% on OpenAI's flagship rates plus constant downward pressure from cheap Chinese open-source models. Back in March this site argued that cheap AI model API pricing could not survive real serving economics. Half right, at best. The floor did not hold, but it fell instead of rising.
So the unit got cheaper and the bill got bigger. That only looks like a contradiction if you assume the unit of consumption stayed still, and it did not. Gartner's 2026 numbers show inference overtaking training spend for the first time in the industry's history, which is what happens when models stop being projects and start being production traffic that runs every minute of every day. On top of that shift, the shape of a single request changed. EY's 2026 analysis priced one agentic interaction at roughly $1.20 against $0.04 for a simple linear workflow, about thirty times the cost for what a budget line still calls "a request." Where the industry is still guessing, in my view, is whether that premium is a transitional inefficiency that better routing and caching will grind away, or the permanent price of letting software think in loops. Nobody has the data to settle it yet, and anyone claiming otherwise is selling something.
Average inference price
$1.16
Per million tokens, 8 August
Time to that low
10 weeks
Measured from 31 May
Agentic coding token use
1,000x
Versus code chat tasks
Inference share of AI cloud
55%
First year above training
That third number is the one that quietly wrecks a forecast. The researchers behind it, a Stanford Digital Economy Lab and Microsoft Research group publishing in April 2026, found the cost sits on the input side rather than the output side: an agent re-reads its own accumulated history into context at every step, so a task that produces four lines of code may have read a small library to get there. Charge that pattern at last year's prices and you still lose, because the workload grew faster than the discount. It also explains why the honest ROI maths on small-business AI agents looked so tight even when the sticker price per million tokens was falling every quarter.
Ten weeks took a million tokens down to $1.16. Your agents burn a thousand times more of them than a chat box ever did. The discount never had a chance.
What The 2026 Numbers Actually Say
Strip out the vendor blogs and the figures that survive come from three places: an investment bank reading a pricing index, an analyst house forecasting cloud spend, and a peer-reviewed measurement of what agents actually consume. Here is the short version worth having on one screen before the next budget review.
| Category | Detail | Insight |
|---|---|---|
| Price index | Average inference sat at $2.04 per million tokens on 31 May 2026 (Jefferies, Silicon Data index) | Unit cost stopped being the binding constraint |
| Spend mix | Gartner puts 2026 AI-optimised IaaS at $42 billion, of which inference takes $23.3 billion | Running models now outspends building them |
| Token appetite | Gartner's March 2026 work found agentic models use five to thirty times more tokens per task than a standard chatbot query | Demand grows faster than prices fall |
| Forecast risk | Models predict their own token consumption at correlations of only 0.39, and systematically guess low | Estimates fail in the expensive direction |
| Visibility | Only 31% of the 512 IT professionals in Flexera's 2026 State of ITAM survey report accurate visibility into AI software spend | Most teams cannot see the bill forming |
| Next year | Gartner sees the same category reaching $66 billion in 2027, with inference at 59% of it | Budget pressure gets worse, not better |
Read the rows together and the pattern is plain. Every line that describes price is improving. Every line that describes consumption, forecasting accuracy or visibility is getting harder to manage. A cost problem you can measure is an engineering problem. A cost problem nobody can see until the invoice arrives is a governance problem, and those get solved much later and much more expensively.
Roughly half of surveyed organisations are already slowing or shrinking AI programmes on cost grounds, which is what a price war looks like from the buyer's side of the invoice.
Friction Points
The uncomfortable part is that most of this spending is not waste in the obvious sense. It is work getting done at a price nobody quoted. Flexera's 2026 survey found 59% of respondents reporting more wasted software spend year over year, and wasted is doing heavy lifting in that sentence, because a retry storm, an over-eager retrieval step and a genuinely useful agent all look identical on a token meter. You cannot tag what you cannot attribute.
Compliance adds its own quiet overhead, and it landed this month. The EU AI Act transparency duties that switched on in August mean disclosure text, logging and audit trails riding along with production traffic, and every one of those is tokens or storage or both. Small per call. Not small at a million calls. And it arrives in the same quarter finance decided to start asking questions, which is either bad luck or a useful forcing function depending on how your quarter is going.
Watch for these before the next invoice cycle:
- Context growth, not call growth. Track average input tokens per task weekly. A flat call count with rising input size is the classic silent doubling.
- Retries billed as work. Failed agent runs consume the full context before they fail (and yes, that includes the one that failed silently last Thursday). Meter them separately or they hide inside your success numbers.
- Model choice by habit. The same task on two frontier models can differ by more than a million tokens. Route by task class, not by whichever model the team liked in January.
- Estimates written by the model. If your capacity plan came from asking an assistant what a workload would cost, assume it guessed low and rebuild the estimate from logged runs.
Key takeaways
Gartner expects worldwide AI spending to reach $2.52 trillion in 2026, up 44% year over year, so the money is not going back in the box.
Cheaper tokens make experiments affordable and production expensive at the same time, and the second effect is larger.
Attribution beats negotiation right now. A per-team, per-workflow token view is worth more this year than another round of vendor discounts.
Stop treating the price sheet as the story. Pull last month's logs, split token spend by workflow rather than by vendor, and find the three workflows carrying the largest input growth. If you cannot produce that split by Friday, that gap is your actual problem, not the price per million tokens.
No comments:
Post a Comment