A demo agent needs a model and a loop. A production agent needs twelve components, and the four teams reliably skip are budget control, observability, permissions and evaluation. Those four are what separate a system you can operate from one you can demonstrate.
The gap between a working prototype and a deployable agent is not model quality. It is everything around the model that nobody needs until real traffic arrives.
What follows is the full inventory, marked by whether it is genuinely required and when to build it.
At a glance
- Four blocks are load-bearing and consistently skipped
- Budget control prevents the single most expensive failure mode
- Observability is not optional; agents fail in ways logs must reconstruct
- Permissions belong outside the model, in the orchestration layer
- Memory is frequently added before it is needed
- Build order matters more than component choice
The twelve blocks
| # | Block | Job | Skip at your peril? |
|---|---|---|---|
| 1 | Model interface | Abstraction over providers, with retries and fallbacks | Required |
| 2 | Orchestration loop | Decides when to call a tool, continue, or stop | Required |
| 3 | Tool registry | Declares available tools, schemas and descriptions | Required |
| 4 | Context assembly | Builds each prompt from instructions, history and retrieved data | Required |
| 5 | Budget control | Caps tokens, tool calls, wall time and spend per run | Load-bearing |
| 6 | Observability | Traces every step with inputs, outputs and timings | Load-bearing |
| 7 | Permissions | Decides what this agent, for this user, may invoke | Load-bearing |
| 8 | Evaluation | Measures task success on a fixed set before deploying | Load-bearing |
| 9 | Output validation | Checks results before they reach the user or a system | Strongly advised |
| 10 | Memory | Persists state across turns or sessions | Often premature |
| 11 | Human checkpoints | Approval gates on consequential actions | Contextual |
| 12 | Multi-agent routing | Delegates between specialised agents | Usually premature |
Blocks 5 through 8 are where production readiness actually lives, and they are invisible in a demo because a demo has no cost ceiling, no debugging need, no untrusted users and no regression risk.
Why is budget control load-bearing?
Because an agent’s worst failure mode is not a wrong answer. It is a loop that never terminates.
Figure 1 — Two ways a run ends
Bounded run
Hard caps on tool calls, total tokens, wall-clock time and spend. Every run terminates one way or another.
A misbehaving agent hits a ceiling, reports failure, and costs a known maximum.
Failure is bounded
Unbounded run
The loop continues while the model keeps proposing actions.
A retry loop or a task the agent cannot complete runs until something external intervenes. Cost is discovered on the invoice.
Failure is unbounded
Four independent caps, not one. Token limits do not catch slow tool calls, and time limits do not catch expensive ones. Each dimension needs its own ceiling, checked every iteration.
What does observability need to capture?
Enough to reconstruct a run without rerunning it.
- Every model call with the assembled prompt, not just the user message
- Every tool call with arguments and the full response
- Every decision point, including what the loop chose not to do
- Cumulative budget at each step, so exhaustion is traceable
- The final outcome and how the run terminated
The assembled prompt is the field teams most often omit and most often need. When an agent behaves oddly, the question is almost always what it actually saw, and reconstructing that from components after the fact is unreliable.
Watch for this
Agent runs are non-deterministic, so a bug that appeared once may not reproduce. Without full traces you are debugging from a description of something that happened rather than a record of it. Build tracing before the first deployment, not after the first incident, because the incident is precisely when the data will be missing.
Where do permissions belong?
In the orchestration layer, evaluated per invocation against the acting user.
Putting them in the prompt makes them advisory. Putting them in the tool implementation scatters the policy across every tool and makes it unauditable. Centralising in orchestration gives one enforcement point and one place to log denials.
The check needs three inputs: which agent is acting, which user it acts on behalf of, and what the requested action is. Agents acting without a user identity are a design smell, because nothing constrains what they may reach.
In what order should you build?
- Model interface, loop, tools, context assembly. The minimum that does anything.
- Budget control. Before the first real traffic, not after the first surprise.
- Observability. Before the first deployment, because it is the debugging substrate.
- Evaluation. Before the second change, so you can tell whether it helped.
- Permissions. Before untrusted users, without exception.
- Output validation. Before results flow into other systems.
- Memory, checkpoints, multi-agent. Only when a measured need appears.
Steps 2 and 3 are the ones that get deferred and should not be. Both are cheap to build early and expensive to retrofit, and both are needed exactly when things go wrong.
Why are memory and multi-agent usually premature?
Because both add substantial complexity to solve problems most agents do not yet have.
Memory is frequently added because it sounds necessary, then holds stale state that degrades behaviour. Many agents work better stateless with retrieval, since a fresh, correctly assembled context beats accumulated history of uncertain relevance.
Multi-agent routing multiplies the failure surface. Every handoff is a place for context to be lost, and latency compounds along the chain. A single agent with good tools handles more than teams expect, and splitting should follow a measured limitation rather than an architectural preference.
Where do teams go wrong?
Building the interesting parts first
Multi-agent orchestration and memory are more engaging to build than budget caps and trace logging. That ordering produces systems that demo well and cannot be operated.
Treating the loop as the product
The orchestration loop is roughly a hundred lines and is not where difficulty lives. Tool design, context assembly and evaluation are the work, and they are where quality is actually determined.
Skipping evaluation because tasks are open-ended
Open-ended does not mean unmeasurable. Fifty representative tasks with human-judged success criteria give you a number, and a number is what makes the next change assessable.
What do experienced teams do differently?
They make every run replayable from its trace.
Given a stored trace, the system can re-execute with the same inputs and a pinned model version, which converts “it did something strange” into a reproducible case. That single capability changes agent debugging from archaeology into engineering.
They also treat the tool registry as the primary product surface. Most agent quality is determined by which tools exist and how they are described, so that is where review attention goes rather than into prompt wording.
A short glossary
- Orchestration loop
- The control flow deciding at each step whether to call a tool, continue reasoning, or stop.
- Context assembly
- The component building each prompt from instructions, history, retrieved data and tool declarations.
- Budget control
- Independent caps on tool calls, tokens, wall time and spend, enforced every iteration.
- Trace
- A complete record of a run, sufficient to reconstruct and ideally replay what happened.
- Human checkpoint
- An approval gate requiring explicit confirmation before a consequential action proceeds.
Key takeaways
- Four components separate demo agents from production ones: budget control, observability, permissions and evaluation.
- An agent’s worst failure is an unbounded loop, so cap tool calls, tokens, time and spend independently.
- Traces must capture the assembled prompt, not just the user message, because that is what you will need.
- Permissions belong in orchestration, evaluated per invocation against the acting user.
- Build budget control and observability before first deployment; both are expensive to retrofit.
- Memory and multi-agent routing are usually premature and add failure surface.
- The orchestration loop is not the hard part; tool design and context assembly are.
Frequently asked questions
What separates a production agent from a prototype?
Four components that a demo never needs: budget control, observability, permissions and evaluation. A prototype has no cost ceiling, no debugging requirement, no untrusted users and no regression risk, so these are invisible until real traffic arrives.
Why does an agent need budget limits?
Because its worst failure is not a wrong answer but a loop that never terminates. Cap tool calls, total tokens, wall-clock time and spend independently, since token limits do not catch slow tool calls and time limits do not catch expensive ones.
What should agent traces record?
Every model call with the fully assembled prompt, every tool call with arguments and full response, each decision point, cumulative budget at each step, and how the run terminated. The assembled prompt is the most commonly omitted and most frequently needed field.
Where should agent permissions be enforced?
In the orchestration layer, checked per invocation against which agent is acting, which user it acts for, and what action is requested. Prompt-level rules are advisory, and per-tool checks scatter policy into an unauditable form.
Do I need memory in my agent?
Less often than assumed. Memory adds complexity and can hold stale state that degrades behaviour. Many agents perform better stateless with retrieval, since a freshly assembled correct context beats accumulated history of uncertain relevance. Add it when a measured need appears.
When should I use multiple agents?
After a single agent with good tools has demonstrably hit a limit. Every handoff loses context and adds latency, multiplying the failure surface. Multi-agent architectures should follow a measured constraint rather than an architectural preference.
How do I evaluate an open-ended agent?
Open-ended does not mean unmeasurable. Assemble fifty representative tasks with human-judged success criteria and score against them. The absolute number matters less than having one, because it is what makes the next change assessable rather than a matter of impression.
What is the right build order?
Model interface, loop, tools and context assembly first, then budget control before real traffic, observability before deployment, evaluation before the second change, permissions before untrusted users, and validation before results reach other systems. Memory and multi-agent last.
References
- Anthropic. Model Context Protocol documentation, covering tool registries and host-level permission enforcement.
- Schick, T., et al. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761.
- NIST. AI Risk Management Framework. Relevant to permission and approval design.
- Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172. Underpins the context assembly component.
