- >55%share of AI-optimized infrastructure spend now consumed by inference, surpassing training for the first time in early 2026
- 280xdrop in cost to query a GPT-3.5-class model, from $20.00 to roughly $0.07 per million tokens between late 2022 and late 2024
- $600Bhyperscaler capex in 2026, up 36% year over year, with about 75% ($450B) tied directly to AI infrastructure
The single most important thing to understand about AI infrastructure economics in 2026 is that “training cost” and “inference cost” are not two flavors of the same expense — they behave like fundamentally different categories of spending, and treating them as one line item is why so many AI budgets blow past their forecasts. Training a model is a capital investment: you spend a large, roughly known amount once (or periodically, for a retrain), and the spending stops when the run finishes. Inference is an operating expense: every single user interaction costs something, it never stops as long as the product is live, and it scales directly with the thing every product team is actually trying to maximize — usage.
The crossover that already happened
For the first time, inference workloads now consume more than 55% of AI-optimized infrastructure spending, and industry trackers put more than $50 billion being spent on inference in 2026 alone — more than is being spent on training frontier models. AI budgets that were roughly 20% inference and 80% training in 2023 have flipped to as much as 85% inference in some organizations by 2026. McKinsey’s own projection extends this further: by 2027, inference is expected to fully overtake training, representing 70-80% of total AI compute and 30-40% of total data center demand by 2030.
| Training | Inference | |
|---|---|---|
| Spending pattern | Bounded, one-time or periodic capital outlay | Continuous, scales with usage — an operating expense |
| Cost driver | Model size, dataset size, number of training runs | Query volume, context length, output length, concurrency |
| Predictability | Estimable in advance to within a reasonable margin | Directly tied to product success — the better you do, the more it costs |
| 2026 share of AI infra spend | Under 45% and falling | Over 55% and rising |
The cost paradox: cheaper per token, more expensive in total
Here is the detail that trips up most cost forecasts: per-token inference has gotten dramatically cheaper, while total inference spend keeps climbing. The cost of querying a model at roughly GPT-3.5’s capability level on the MMLU benchmark fell from about $20.00 per million tokens in late 2022 to roughly $0.07 per million tokens by late 2024 — more than a 280-fold reduction in under two years. If you only looked at that number, you’d expect total inference spend to be falling. It’s doing the opposite, because volume grew even faster than price fell. Agentic workloads in particular — an agent that calls a model dozens of times to complete one task, rather than a single user query — multiply token volume in a way that per-token pricing improvements can’t offset.
- 1. Price fallsPer-million-token cost drops roughly 280x for GPT-3.5-class capability between late 2022 and late 2024.
- 2. Usage expandsCheaper inference makes previously uneconomical use cases viable, so query volume rises.
- 3. Agentic multiplicationAgents call models many times per task rather than once per user query, multiplying token volume further.
- 4. Net spend risesVolume growth outpaces price decline, and total inference spend climbs even as unit economics improve.
Why inference spend is rising even though inference is getting cheaper
This is the Jevons paradox playing out in AI infrastructure: efficiency gains that expand usage enough to increase total consumption, not reduce it.
Why this changes how you should budget
Industry analysis puts inference at 80-90% of the lifetime cost of a production AI system, because it runs continuously for as long as the product exists, while training is a bounded event that happens once or periodically. A team that budgets for a big training run and treats inference as a rounding error is budgeting for the smaller and more predictable half of its actual lifetime AI spend. The practical implication is that cost engineering effort — quantization, caching, batching, model routing by query difficulty — pays off disproportionately on the inference side, precisely because it’s the side that compounds indefinitely rather than the side that happens once.
Budget training as capex Bounded
Estimate the run cost in advance, treat overruns as a project-management problem, and expect the spending to stop when the run finishes.
Budget inference as opex tied to growth Unbounded
Model inference cost per unit of usage, then multiply by your growth projection — a flat inference budget number is close to meaningless once usage is the actual driver.
Treat them as one combined “AI compute budget” line Miscalibrated
Conflating a bounded one-time cost with an unbounded scaling cost produces forecasts that are wrong in both directions at once.
Common mistakes
Common mistake
Forecasting next year’s AI infrastructure budget as a percentage increase over this year’s combined training-plus-inference number. Because the two scale on entirely different drivers, this produces a forecast that’s structurally wrong regardless of the percentage chosen.
Common mistake
Celebrating a falling per-token price as evidence that inference costs are under control. Total spend is what hits the budget, and volume — especially agentic, multi-call volume — has been growing faster than price falls for several years running.
What worked
Teams that got ahead of this split their AI cost tracking into the two categories explicitly from day one, modeling inference cost per unit of product usage rather than as a flat monthly number, which let them see cost overruns coming from usage growth rather than discovering them after the invoice.
Glossary
- Training cost
- The capital expense of running a model training job — bounded, largely predictable in advance, and it stops when the run completes.
- Inference cost
- The operating expense of running a trained model to answer queries — continuous, tied directly to usage volume, and never stops while the product is live.
- Jevons paradox (as applied to AI inference)
- The pattern where falling per-unit cost of a resource increases total consumption of it by enough to raise overall spend, rather than lowering it.
- Agentic token multiplication
- The effect where an AI agent calling a model multiple times to complete a single task multiplies total token volume well beyond a simple one-query-per-user pattern.
Key takeaways
- Inference now consumes over 55% of AI-optimized infrastructure spending, surpassing training for the first time, with over $50 billion spent on inference in 2026 alone.
- Training is a bounded capital investment; inference is an unbounded operating expense that scales directly with usage — treating them as one budget line produces structurally wrong forecasts.
- Per-token inference cost fell roughly 280-fold for GPT-3.5-class capability between late 2022 and late 2024, yet total inference spend kept climbing because usage volume grew even faster.
- Agentic workflows multiply token volume by calling a model many times per task, which is a major driver of rising total inference spend even as per-token pricing keeps improving.
- Industry analysis puts inference at 80-90% of a production AI system’s lifetime cost, since it runs continuously while training happens once or periodically.
- McKinsey projects inference will represent 70-80% of total AI compute and 30-40% of total data center demand by 2030, continuing the crossover already underway.
- Effective cost engineering — quantization, caching, batching, query routing by difficulty — pays off disproportionately on the inference side because that’s the cost that compounds indefinitely.
FAQs
Does inference really cost more than training now?
Yes, in aggregate. Inference workloads now consume over 55% of AI-optimized infrastructure spending, having surpassed training for the first time in early 2026, with more than $50 billion spent on inference in 2026 alone.
Why is total inference spend rising if per-token prices are falling?
Because usage volume is growing faster than price is falling. Per-token cost for GPT-3.5-class capability dropped roughly 280-fold between late 2022 and late 2024, but query volume — especially from agentic workflows calling a model many times per task — has grown even faster, pushing total spend up.
Why shouldn’t training and inference be budgeted as one combined line item?
Because they behave completely differently: training is a bounded, one-time or periodic capital expense that stops when a run finishes, while inference is a continuous operating expense that scales directly with product usage and never stops while the product is live.
What share of a production AI system’s lifetime cost is inference?
Industry analysis puts it at roughly 80-90%, since inference runs continuously for the life of the product while training happens once or periodically, making inference the dominant cost over any meaningful time horizon.
How much are hyperscalers spending on AI infrastructure in 2026?
Hyperscaler capital expenditure reached about $600 billion in 2026, a 36% increase over 2025, with roughly 75% (about $450 billion) tied directly to AI infrastructure.
What is causing inference costs to keep rising despite efficiency gains?
Agentic workloads are a major driver — an AI agent calling a model dozens of times to complete a single task multiplies token volume in a way that per-token price improvements can’t offset, on top of general usage growth from cheaper access.
When will inference fully overtake training in AI compute demand?
McKinsey projects that by 2027 inference will represent 70-80% of total AI compute, and by 2030 it will account for 30-40% of total data center demand, continuing a crossover that already began in early 2026.
How should a team budget for AI inference costs correctly?
By modeling inference cost per unit of product usage and multiplying by growth projections, rather than setting a flat monthly inference budget — since usage, not a fixed number, is the actual driver of total spend.
References
- Byteiota — “AI Inference Costs: 55% of Cloud Spending in 2026”
- Introl — “AI Inference vs Training Infrastructure Economics”
- Spheron — “AI Inference Cost Economics in 2026: GPU FinOps Playbook”
- Telnyx — “AI training vs inference: the 2026 economics split”
Related reading on The Tech Trends: NVIDIA vs AMD vs Custom Silicon, Renting vs Owning GPUs, GPU Shortage Economics, Prompt Caching and Batching, and Quantization Explained.
