Agent spend is unbounded by default because the loop, not you, decides how many steps a run takes. Four hard caps make failure bounded; six efficiency measures reduce the cost of success. Start with the caps, because they turn an open-ended liability into a known maximum.
A conventional API call has a knowable cost. An agent run does not: it costs whatever the loop decides, and the loop is driven by a model that has no idea what anything costs.
That is the whole problem. Every control below is a way of putting a boundary around a process that has none by construction.
| # | Control | Type | Typical impact |
|---|---|---|---|
| 1 | Cap tool calls per run | Hard limit | Bounds the worst case |
| 2 | Cap total tokens per run | Hard limit | Bounds the worst case |
| 3 | Cap wall-clock time | Hard limit | Catches slow tools budgets miss |
| 4 | Cap spend per run and per user | Hard limit | Bounds abuse and bugs |
| 5 | Route by difficulty | Efficiency | Large, where traffic is mixed |
| 6 | Order prompts for cache reuse | Efficiency | Large, and changes no output |
| 7 | Prune conversation history | Efficiency | Large in long sessions |
| 8 | Trim tool declarations per task | Efficiency | Moderate, improves accuracy too |
| 9 | Detect and break loops | Efficiency | Catches the worst outliers |
| 10 | Summarise tool output before context | Efficiency | Moderate to large |
Controls 1 to 4 do not reduce average cost at all. They exist so that when something goes wrong, it goes wrong for a known maximum rather than until someone notices.
Why do four separate caps?
Because each catches a failure the others miss.
Figure 1 — What each cap catches
- CAP 1Tool callsCatches loops. Misses one call that returns an enormous payload.
- CAP 2TokensCatches context bloat. Misses cheap calls that are merely slow.
- CAP 3Wall timeCatches hanging tools. Misses fast expensive model calls.
- CAP 4SpendThe backstop, expressed in the unit that actually matters
Check all four every iteration, not at the end. A cap evaluated only on completion is not a cap; it is a post-mortem. The loop must test its budget before deciding to continue.
Which efficiency measure pays most?
Prompt ordering for cache reuse, because it changes no output and costs a few minutes.
If your prompt begins with a session ID or timestamp, the shared prefix differs on every call and caching never engages. Moving static instructions ahead of variable content converts a cache that never hits into one that hits nearly always, with identical results.
Routing by difficulty comes second. Serving simple lookups from a fast model and reserving expensive reasoning for queries that need it is the largest structural saving available where traffic is genuinely mixed.
Watch for this
Cost per run is a misleading metric and cost per successful task is the honest one. A cheaper model that fails more often raises total cost per outcome while every per-run dashboard improves. Divide spend by successful completions, not by attempts, or you will optimise your way into a more expensive system that looks cheaper.
Where does the spend actually go?
Rarely where teams expect. Instrument before optimising.
- Repeated system prompt on every iteration of every run, which is often the single largest line
- Unpruned conversation history, growing without bound across a session
- Tool declarations for tools the current task will never use
- Raw tool output pasted into context, including fields nothing reads
- Retries after failures, which multiply everything above
The fourth deserves attention because it is easy to fix. A tool returning a full API response often contributes thousands of tokens where a summarised subset would serve, and the summarisation can happen in your code without a model call.
How do you break loops?
- Hash tool name plus arguments on every call and count repeats within the run.
- Break at the second identical failure. A call that failed twice with identical arguments will fail again.
- Detect alternation as well as repetition, since agents oscillate between two tools as readily as repeating one.
- Return a terminal message the agent can relay, rather than failing silently into the cap.
Loop detection is cheap and catches the outliers that dominate a cost distribution. A handful of runaway runs a month can exceed the entire cost of normal traffic.
What should you monitor?
| Signal | What it tells you |
|---|---|
| Cost per successful task | The only figure that is economically meaningful |
| Budget exhaustion rate | How often runs fail expensively; should be near zero |
| Step count distribution | Whether the agent is working harder for the same result |
| Cache hit rate | Whether prompt ordering is still intact after deploys |
| Cost distribution tail | Whether outlier runs dominate the total |
Watch the tail, not the mean. Agent cost distributions are heavily skewed, so the average conceals the small number of runs that account for most of the bill.
Where do teams go wrong?
Optimising the model before the plumbing
Downgrading to a cheaper model trades quality for cost while an uncacheable prompt and unpruned history sit untouched. Fix the free things first, because they cost nothing in quality.
Setting caps too generously to avoid complaints
A limit set so high it never triggers provides no protection. Set it near the observed upper bound of legitimate runs and treat exhaustion as a signal to investigate rather than a threshold to raise.
Measuring cost without measuring success
Cost figures divorced from outcomes lead directly to cheaper systems that work worse. The two numbers only mean something together.
What do experienced teams do differently?
They attribute cost per run, per user and per tool from day one.
When spend rises, the question is immediately which tool, which task type or which user changed, rather than a general investigation. Attribution built into tracing costs nothing extra, since the traces already record every call.
They also set a per-user daily ceiling separately from the per-run cap. A single run bounded at a sensible limit can still be invoked hundreds of times by a bug or an aggressive integration, and only the second control catches that.
A short glossary
- Hard cap
- An absolute limit terminating a run regardless of state, checked every iteration.
- Cost per successful task
- Total spend divided by successful outcomes rather than by attempts.
- Budget exhaustion
- A run ending because it hit a cap rather than completing its task.
- Prefix caching
- Reuse of the processed opening of a prompt across calls sharing it byte-for-byte.
- Loop detection
- Identifying repeated or alternating identical calls within a run and terminating early.
Key takeaways
- Agent spend is unbounded by default because the loop decides run length, not you.
- Four independent caps are needed: tool calls, tokens, wall time and spend, checked every iteration.
- Caps do not reduce average cost; they make failure cost a known maximum.
- Prompt ordering for cache reuse is the highest-return efficiency measure and changes no output.
- Cost per successful task is the honest metric; cost per run rewards cheap failure.
- Loop detection catches the outlier runs that dominate a skewed cost distribution.
- Set a per-user daily ceiling as well as a per-run cap, since bounded runs can be invoked unboundedly.
Frequently asked questions
Why do agent costs run away?
Because the orchestration loop decides how many steps to take, and the model driving it has no notion of cost. Without explicit caps a run continues while the model keeps proposing actions, so a retry loop or an impossible task runs until something external intervenes.
Why do I need four separate caps?
Each catches a failure the others miss. Tool-call limits catch loops but not one enormous payload. Token limits catch context bloat but not slow calls. Time limits catch hanging tools but not fast expensive ones. Spend is the backstop in the unit that matters.
What is the cheapest cost saving available?
Reordering prompts so static content precedes variable content, enabling prefix cache reuse. It takes minutes, changes no output whatsoever, and can convert a cache that never hits into one that hits on nearly every call.
Should I use a cheaper model to control cost?
Only after the free measures are exhausted. Downgrading trades quality for spend, while uncacheable prompts, unpruned history and untrimmed tool declarations cost nothing in quality to fix. Route by difficulty rather than downgrading everything uniformly.
Why is cost per run misleading?
Because it ignores failure. A cheaper configuration that fails more often can raise total cost per outcome while every per-run figure improves. Dividing by successful completions rather than attempts is what makes the comparison economically honest.
Where does agent spend usually go?
Into repeated system prompts on every iteration, unpruned conversation history, tool declarations for tools the task never uses, raw tool output pasted wholesale into context, and retries that multiply all of the above. Instrument before optimising, since intuition is usually wrong.
How do I stop runaway loops?
Hash tool name and arguments on each call, count repeats within the run, and break at the second identical failure. Detect alternation as well as repetition, since agents oscillate between two tools as readily as repeating one.
Should I watch mean cost or the distribution?
The distribution, particularly the tail. Agent costs are heavily skewed, so a handful of runaway runs can exceed the cost of all normal traffic while the mean looks unremarkable. Monitor the tail and the budget exhaustion rate together.
References
- Kwon, W., et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180. Background on prefix sharing in serving.
- Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172. Why trimming context helps quality as well as cost.
- Anthropic. Model Context Protocol documentation, covering tool declaration structure.
