A latency cascade is what happens when one slow step in a multi-agent chain forces every step after it to wait, and every retry it triggers multiplies that wait. The fix is not a faster model. It is bounding how long any single step can hold up the rest of the run, and refusing to let failures propagate as retries by default.
A single agent call that takes four seconds feels acceptable. Chain five of them sequentially and the run takes twenty seconds before any retries. Add a retry on the one call that occasionally times out, and a run that should take five seconds now sometimes takes forty. Nobody designed for forty seconds. It emerged from the composition.
At a glance
- Latency in a chain adds; latency from retries multiplies
- A single slow or hung step can stall an entire multi-agent run
- Sequential architectures are the most exposed; parallel fan-out is the primary defence
- Default retry behaviour is the most common hidden cause of cascades
- Per-step timeouts and circuit breakers convert an open-ended wait into a bounded one
- The fix is architectural, not a bigger model or faster inference
Why does latency compound instead of just adding up?
| Source | Effect on total latency | Why it is worse than it looks |
|---|---|---|
| Sequential agent calls | Additive | Five 4-second calls in series is a 20-second floor before anything goes wrong |
| A single retry on failure | Multiplicative on that step | One retry doubles that step’s worst case; two retries triple it |
| Retries nested inside a chain | Multiplicative on the whole run | A retried step delays every step after it, not just itself |
| Timeouts set too generously | Raises the ceiling on every stall | A hung call waits the full timeout before anything downstream can react |
| Fan-out with no concurrency limit | Can exceed additive cost | Unbounded parallel calls compete for the same rate limits and slow each other down |
| Tool calls to external APIs | Variable and outside your control | The slowest external dependency sets the floor for any step depending on it |
Rows two and three are where most production incidents come from. Sequential latency is visible in a demo — you can watch it accumulate. Retry-driven latency is invisible until a dependency degrades, at which point every retry attempt compounds the delay and a system that felt fast in testing suddenly does not.
How does one slow agent become a system-wide stall?
Figure 1 — A three-step chain under normal load vs. one slow dependency
Normal conditions
Step A: 2s. Step B: 3s. Step C: 2s. Total: 7 seconds, matching the demo everyone signed off on.
Each step returns promptly and the next step starts immediately. Nothing is waiting on anything unexpected.
7 seconds, as designed
Step B’s dependency degrades
Step B now takes 25 seconds instead of 3. It has a retry configured, so on timeout it retries once, adding another 25-second wait before it gives up and returns an error.
Step A and Step C are unaffected individually, but the total run time is now roughly 52 seconds, and Step C — which had nothing to do with the problem — is what the user sees stall.
52 seconds, from one slow dependency
The retry did not fix anything here; it just paid for the failure twice before giving up. Without a hard ceiling on how long Step B is allowed to hold up the chain, its problem becomes the whole run’s problem, and the component that looks broken to the user (Step C, or the run itself) is not the component that actually failed.
This is the core mechanism behind almost every latency cascade: a local slowdown becomes a global one because nothing in the architecture stops it from propagating. The chain has no circuit breaker, so it pays the full cost of the failing step before anything downstream gets a chance to react.
What are the four places cascades start?
- An external API that degrades without an outage. Still returning 200s, just slower. Nothing alerts on this until a timeout fires, by which point the chain has already absorbed the cost.
- A retry policy applied uniformly. Retrying every failure the same number of times treats a transient network blip and a genuinely overloaded dependency identically, multiplying delay in exactly the case where retrying makes it worse.
- Unbounded fan-out. Dispatching to every available tool or sub-agent at once without a concurrency cap can create contention on a shared rate limit, slowing down calls that would have been fast in isolation.
- A model call inside a loop with no step cap. An agent re-planning repeatedly because it keeps failing the same check can turn a bounded task into an effectively unbounded one, with each iteration adding its own latency.
Notice that none of these four require anything to be technically broken. Each component can be behaving within its own individual spec and the composition still produces a run that takes ten times longer than expected. That is what makes cascades hard to catch in isolated component testing — they are a property of the chain, not of any one link in it.
How do you actually stop a cascade?
| Technique | What it bounds | Trade-off |
|---|---|---|
| Per-step timeout | Maximum wait for any single call | A call that would have eventually succeeded gets cut off early |
| Circuit breaker | Repeated calls to an already-failing dependency | Requires tracking failure state, adding a small amount of complexity |
| Bounded, selective retries | How many times a multiplicative delay can recur | Some transient failures that a blind retry would have fixed now surface as errors |
| Concurrency limits on fan-out | Contention between parallel calls | Slightly lower peak throughput in exchange for predictable per-call latency |
| Step or iteration caps on agent loops | How long an agent can keep re-planning | A genuinely hard task may need more iterations than the cap allows |
| Async fan-out with a join | Sequential accumulation of independent steps | Requires the steps to genuinely have no dependency on each other |
The trade-off column matters more than the technique column. Every one of these bounds costs something — a call cut off early, an error surfaced instead of silently retried, throughput given up for predictability. That is the correct trade. An unbounded system trades a rare catastrophic cascade for a marginally lower rate of individually annoying early cutoffs, and the catastrophic version is worse in every case that matters.
Watch for this
A circuit breaker that trips and never resets is as damaging as no circuit breaker at all — it just fails a different way. Pair every circuit breaker with a half-open retry policy that periodically tests whether the dependency has recovered, and log every state transition. A breaker stuck open silently degrades the whole system into an error path that nobody notices until someone asks why a feature that used to work has been failing for three days.
Where does parallelism actually help, and where does it not?
It helps enormously for genuinely independent steps and does nothing for steps that depend on each other’s output.
If three sub-agents each retrieve different information that a fourth agent will later combine, running those three concurrently turns an additive 9 seconds into roughly 3 — whichever one is slowest. If a chain is sequential because step two genuinely needs step one’s output, no amount of parallelism removes that dependency; the only lever left is reducing the latency of each individual step or removing steps that are not load-bearing.
The design implication is to actively look for false sequentiality — steps that were written to run one after another out of convenience, not because of a real data dependency. Restructuring those into a fan-out with a join is usually the single largest latency win available, well ahead of trying to make any individual model call faster.
Where do teams go wrong?
Setting one retry policy for every call
Retrying is appropriate for a genuinely transient failure and actively harmful against a dependency that is overloaded, where the retry adds load to the exact thing that is already struggling. Differentiate by failure type — timeout versus rate limit versus server error — rather than applying a blanket policy.
Measuring latency only at the run level
An aggregate “average run took 8 seconds” hides a cascade completely if it happens on 2% of runs. Per-step latency, tracked individually and with percentiles rather than averages, is what actually surfaces a step that occasionally takes ten times longer than the others.
Treating timeouts as a safety net instead of a design decision
A timeout set generously “to be safe” simply raises the ceiling on how long a stall is allowed to last. The right timeout is set from the actual distribution of that step’s latency, not from a round number that felt conservative.
What do experienced teams do differently?
They budget latency the same way they budget cost — as an explicit allowance attached to the run, not an emergent property they discover after deployment.
Concretely, that means every step in a chain has a defined maximum contribution to total run time, agreed before the chain is built, with a circuit breaker or fallback for the case where a step cannot meet it. When total elapsed time approaches the run’s overall budget, remaining optional steps are skipped rather than allowed to push the run over a deadline that a downstream consumer is actually waiting on.
They also treat a slow success differently from a fast one in monitoring. A dependency that returns correct answers ten times slower than usual is a leading indicator of an outage that has not happened yet, and treating latency percentiles as an alerting signal — not just an error rate — is what catches a cascade forming before it takes the whole chain down.
A short glossary
- Latency cascade
- A delay in one step of a multi-agent chain that propagates and compounds through every subsequent step.
- Circuit breaker
- A control that stops calling an already-failing dependency after a threshold, instead failing fast or falling back.
- Fan-out
- Dispatching independent work to multiple agents or tool calls concurrently instead of one after another.
- False sequentiality
- Steps built to run one after another despite having no actual data dependency requiring that order.
- Latency budget
- An explicit maximum time allowance for a run, allocated across its steps in advance.
Key takeaways
- Sequential latency adds; retry-driven latency multiplies, and the second is what causes real incidents.
- One slow dependency can stall an entire chain even when every component is individually within spec.
- Circuit breakers, per-step timeouts, and bounded retries convert open-ended waits into predictable ones.
- A blanket retry policy is often the hidden cause of cascades, especially against an overloaded dependency.
- Parallelise genuinely independent steps; false sequentiality is usually the largest available latency win.
- Measure per-step latency percentiles, not run-level averages, or a rare cascade stays invisible.
- Treat latency as an explicit budget attached to the run, allocated across steps before anything is built.
Frequently asked questions
What causes a latency cascade in a multi-agent system?
A slow or failing step that the chain has no mechanism to bound, combined with a retry policy that pays for the delay multiple times before giving up. The chain absorbs the full cost sequentially, and every step after the slow one waits for it regardless of whether they were involved in the problem.
Does adding retries make a system more reliable?
Only against genuinely transient failures. Against an overloaded dependency, retries add load to the exact component that is struggling and multiply the delay for every caller waiting on it. Differentiate retry policy by failure type rather than applying one blanket rule.
How do I stop one slow step from stalling the whole chain?
Set a per-step timeout based on that step’s actual latency distribution, pair it with a circuit breaker that stops calling a dependency once it is clearly failing, and define a fallback or skip path for when the timeout is hit rather than letting the run wait indefinitely.
Does parallelising agent calls always reduce latency?
Only for steps that are genuinely independent. Steps with a real data dependency cannot be parallelised regardless of architecture. The highest-value fix is usually finding steps that were built sequentially out of convenience rather than necessity and restructuring them into a fan-out.
Why do average latency metrics miss cascades?
Because a cascade affecting a small percentage of runs is invisible in an average. Per-step latency tracked with percentiles surfaces a step that occasionally takes far longer than usual, which is exactly the signal a run-level average smooths away.
What happens if a circuit breaker never resets?
The system silently degrades into a permanent error or fallback path for that dependency, and nobody notices until someone asks why a working feature stopped working. Always pair a breaker with a half-open state that periodically tests recovery, and log every transition.
Should every agent call have the same timeout?
No. A timeout set from the actual latency distribution of that specific step is far more useful than one round number applied everywhere. A generous blanket timeout simply raises the ceiling on how long any single stall is allowed to last.
How should I think about latency budgets across a chain?
The same way you budget cost — as an explicit allowance agreed before the chain is built, allocated across steps, with optional steps skipped once elapsed time approaches the overall deadline rather than letting the run exceed what a downstream consumer is actually waiting for.
References
- Nygard, M. Release It! Design and Deploy Production-Ready Software. The circuit breaker pattern this article’s mitigations are built on.
- Anthropic. Model Context Protocol documentation, covering the tool-call layer where much of this latency originates.
- OpenTelemetry. Traces and spans, the instrumentation needed to see per-step latency rather than only run-level totals.
