Task success rate is a lagging aggregate that tells you an agent failed without telling you where. Decompose it into nine measurable parts, and two of them — tool error rate and step count distribution — move before users notice anything. Those two are the ones worth alerting on.
A single success percentage is the agent equivalent of monitoring a web service by whether the homepage loads. It confirms a problem exists and offers nothing about its location.
Agents fail across several independent dimensions, and each needs its own number.
| # | Metric | Answers | Type |
|---|---|---|---|
| 1 | Task success rate | Did the run achieve its goal? | Lagging |
| 2 | Tool error rate | What share of tool calls fail? | Leading |
| 3 | Step count distribution | How many steps does a run take? | Leading |
| 4 | Tool selection accuracy | Was the right tool chosen? | Diagnostic |
| 5 | Budget exhaustion rate | How often do runs hit a cap? | Leading |
| 6 | Recovery rate | After an error, does the run still succeed? | Diagnostic |
| 7 | Escalation rate | How often does it hand off to a human? | Diagnostic |
| 8 | Cost per successful task | What does a win actually cost? | Lagging |
| 9 | Time to first action | How long before anything happens? | Diagnostic |
Cost per successful task, not cost per run, is the number that matters commercially. A cheap agent that fails half the time costs more per outcome than an expensive one that works, and per-run cost hides that completely.
Why do step count and tool error rate lead?
Because both degrade before success rate does, while the agent is still compensating.
Figure 1 — What a shifting step distribution means
Stable distribution
Most runs complete in a consistent narrow band of steps, with a small tail.
The agent is finding paths efficiently, and the tail is genuinely harder tasks rather than confusion.
Healthy
Rightward drift
The median creeps up and the tail thickens, while success rate holds steady.
The agent is working harder for the same outcome: retrying, exploring, recovering. Success is being bought with budget, and the trend ends at the cap.
Degrading
This is the single most useful agent metric. Success rate is flat right up until it collapses; step count shows the pressure building weeks earlier, and it costs nothing to compute from traces you already store.
Tool error rate behaves similarly. An API changing its response shape, a rate limit tightening, or a description drifting all raise error rate first. The agent retries and often still succeeds, so success rate stays flat while cost and latency climb.
How do you measure tool selection accuracy?
By labelling, since no automatic signal exists.
Sample fifty runs, and for each tool call have a human record whether that was the correct tool given the state at that moment. It is tedious and it is the only way to separate “chose wrongly” from “chose correctly and the tool failed”, which need entirely different fixes.
Do this quarterly and after any change to tool descriptions or the tool set. It is not a continuous metric; it is a periodic audit.
Watch for this
A rising recovery rate looks like resilience and often signals the opposite. If more runs are succeeding only after an error, the underlying error rate is climbing and the agent is masking it. Always read recovery rate next to tool error rate, never alone, or you will congratulate a system that is quietly deteriorating.
What should trigger an alert?
- Budget exhaustion rate above a small threshold. Runs hitting caps are runs that failed expensively, and the rate should be near zero.
- Step count median shifting beyond its normal band, which indicates degradation before outcomes move.
- Tool error rate rising on any individual tool, which usually means an upstream change.
- Cost per successful task climbing while success rate is flat, the signature of an agent working harder for the same result.
Notice that task success rate is absent from the alert list. It is the metric to report and the wrong one to page on, because by the time it moves the problem has been running for some time.
How do you define success at all?
This is harder than the measurement and is where most agent evaluation stalls.
For agents producing structured output or making state changes, success is checkable: the record was created, the value is correct, the schema validates. For open-ended tasks it needs human judgement against explicit criteria, which means fifty labelled tasks rather than an automated score.
The practical compromise most teams reach: define success narrowly enough to be checkable even if that undercounts partial wins. A conservative measurable definition beats a generous unmeasurable one, because only the first supports comparison over time.
Where do teams go wrong?
Reporting averages over mixed traffic
Easy and hard tasks aggregate into a number that means nothing. If the traffic mix shifts, the metric moves without any change in capability. Segment by task type and report separately.
Measuring the agent instead of the system
Users experience the whole pipeline, including retrieval, tools and validation. An agent with excellent tool selection against a broken retrieval layer still fails, and agent-only metrics will not show it.
Optimising cost per run
Cutting cost per run by using a weaker model can raise cost per successful task, because failures multiply. Always divide by successes rather than by attempts.
What do experienced teams do differently?
They compute metrics from stored traces rather than instrumenting separately.
If traces capture every model call, tool call and outcome, all nine metrics are queries over that data. This means new metrics can be computed retroactively over historical runs, which is exactly when you want them, and it removes the drift between what is logged and what is measured.
They also track metrics per tool as well as per agent. A single failing tool can drag aggregate reliability down while every other part of the system is healthy, and only per-tool breakdown makes that visible in one look.
A short glossary
- Leading metric
- A measure that degrades before user-visible outcomes do, suitable for alerting.
- Recovery rate
- The share of runs that succeed despite encountering at least one error along the way.
- Budget exhaustion
- A run terminating because it hit a cap on steps, tokens, time or spend rather than completing.
- Cost per successful task
- Total spend divided by successful outcomes rather than by attempts.
- Step count distribution
- The spread of how many iterations runs take, whose rightward drift signals degradation.
Key takeaways
- Task success rate is a lagging aggregate; it confirms failure without locating it.
- Step count distribution and tool error rate degrade before outcomes do, which makes them the alerting metrics.
- A rising median step count with flat success means the agent is buying success with budget.
- Recovery rate must be read alongside error rate, or masking looks like resilience.
- Cost per successful task is the commercial number; cost per run hides failure multiplication.
- Tool selection accuracy needs periodic human labelling, since no automatic signal exists.
- Compute metrics from stored traces so new measures can be applied to historical runs.
Frequently asked questions
What is the most useful single agent metric?
Step count distribution. Success rate stays flat until it collapses, whereas a rightward drift in median steps shows the agent working harder for the same outcome weeks earlier. It also costs nothing to compute from traces you already store.
Why is task success rate a poor alerting metric?
Because it lags. By the time success rate moves, the underlying degradation has been running for some time while the agent compensated through retries and extra steps. Alert on tool error rate, step count and budget exhaustion instead.
How do I measure tool selection accuracy?
By human labelling, since no automatic signal exists. Sample fifty runs and record for each tool call whether it was the right choice given the state at that moment. Run it quarterly and after any change to tool descriptions or the tool set.
Is a high recovery rate good?
Not on its own. If more runs succeed only after encountering errors, the underlying error rate is climbing and the agent is masking it. Always read recovery rate alongside tool error rate, or a deteriorating system will look resilient.
Should I measure cost per run or per success?
Per success. A cheaper model that fails more often can raise total cost per outcome even while cost per run falls. Dividing by successful completions rather than attempts is what makes the comparison economically meaningful.
How do I define success for open-ended tasks?
Narrowly enough to be checkable, even if that undercounts partial wins. A conservative measurable definition supports comparison over time; a generous unmeasurable one does not. For genuinely subjective outcomes, use fifty human-labelled tasks with explicit criteria.
Should metrics be aggregated across all traffic?
No. Mixing easy and hard tasks produces a number that moves when traffic composition shifts, independent of any capability change. Segment by task type and report each separately, then look at aggregates only for high-level reporting.
What does a rising budget exhaustion rate mean?
That runs are failing expensively. Hitting a cap means the agent could not complete within its allowance, so the rate should sit near zero. Any sustained rise indicates either task difficulty increasing or the agent losing efficiency, and both warrant investigation.
References
- Anthropic. Model Context Protocol documentation, covering tool invocation structure that traces record.
- Liang, P., et al. Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
- Dettmers, T., et al. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. Source for caution on trusting aggregate benchmark scores.
