| Discipline | What it measures or governs | Primary owner |
|---|---|---|
| Human-agent KPI design | Whether the combined human-agent team is more productive than either alone | COO, HR, department heads |
| Agent operations (AgentOps) | Whether agents in production are running, healthy, and safely rolled back or decommissioned when needed | ML platform engineering, on-call rotation |
A different question than “is this agent any good”
Most of what has been written about managing AI agents in the enterprise focuses on a single question: is the agent doing a good job? That is a real and important question, and our earlier guide on managing the silicon workforce and designing KPIs for human-agent teams covers it well, laying out metrics like verification effort and human-agent synergy scores. This piece deliberately does not repeat that ground.
The question here is different and, for anyone who has actually run agents in production past the pilot stage, arguably more urgent day to day: when an agent’s underlying model gets updated overnight and its behavior shifts, who gets paged? When an agent silently burns through ten times its normal token budget, who catches it before the invoice arrives? When an agent needs to be permanently retired because the business process it supported no longer exists, what is the actual checklist for doing that safely? This is the domain of agent operations, or AgentOps: the operational runbook discipline of running a fleet of AI agents the way a site reliability engineering team runs a fleet of servers.
Why AgentOps became its own discipline
By 2026, autonomous AI agents have moved from demos to core infrastructure at a majority of large enterprises, with more than half of surveyed companies reporting AI agents running in actual production workloads rather than sandboxed pilots. Enterprises now run a double-digit number of distinct agents on average, spanning customer service resolution, financial workflow execution, IT incident triage, and internal research and reporting tasks. A meaningful share of these agents operate largely in isolation from one another, without shared tooling for monitoring, versioning, or cost control, which is precisely the fragmentation problem AgentOps as a discipline is meant to solve.
Unlike traditional software, agents built on large language models are probabilistic rather than deterministic, and their behavior can change even when no one on the engineering team has touched the code, simply because the underlying model provider shipped an update. This single fact is why agent operations cannot simply reuse existing DevOps tooling wholesale; it needs its own practices for drift detection, rollback, and cost containment layered on top of familiar operational disciplines.
The agent incident lifecycle
A simple flow diagram traces an agent incident from automated anomaly detection, through paging the on-call engineer, to triage and rollback to the last known-good agent version, and finally to a post-incident review that feeds back into the agent’s evaluation suite before it is re-enabled.
On-call rotations for agent incidents
The first pillar of agent operations is treating agent incidents the way a mature engineering organization treats any production incident: with a defined on-call rotation, clear severity levels, and a runbook rather than ad hoc firefighting. Unlike a traditional application outage, an “agent incident” often does not look like a crash; the agent is still running and still producing output, but that output has drifted into wrong, unsafe, or unusually expensive territory. This makes agent incidents harder to detect automatically and more dependent on continuous evaluation against a golden dataset of known-correct behavior.
Effective on-call rotations for agent operations pair a traditional engineer with someone who understands the specific business logic the agent is automating, since diagnosing whether an agent’s output is subtly wrong often requires domain knowledge that a pure infrastructure engineer will not have. Escalation severity is typically tiered by the blast radius and reversibility of the agent’s actions: an agent that only drafts content for human review can tolerate a slower response than one with standing authority to send money or modify customer records.
| Severity tier | Example trigger | Target response time |
|---|---|---|
| Critical | Agent takes an irreversible action outside approved permissions, such as an unauthorized financial transaction | Immediate, page on-call |
| High | Sharp spike in error rate or hallucination frequency detected by automated evaluation | Under 30 minutes |
| Medium | Cost per run exceeds budget threshold without a corresponding output quality gain | Same business day |
| Low | Gradual model drift detected against golden dataset benchmarks | Next scheduled review cycle |
Deployment and rollback procedures for agents
Because agent behavior depends on both the code around it and the underlying model version it calls, agent deployment has two axes of versioning that traditional software deployment does not: the orchestration code, and the model or prompt configuration. Mature agent operations teams version both together, so that a rollback restores not just the code but the exact model version and prompt template that were in place at a known-good point in time. Without this discipline, teams often discover that “rolling back” a code deployment does not fix a behavior regression, because the actual cause was an upstream model update that the rollback did not touch.
A practical deployment pattern that has become standard in mature AgentOps setups is canary release for agents: a new agent version handles a small, monitored percentage of real traffic while the previous version continues handling the majority, with automated comparison of output quality and cost between the two before a full cutover. This mirrors canary deployment in traditional software but adds agent-specific checks, particularly around hallucination rate and task completion accuracy, that a standard software canary would not evaluate.
Common mistake
Treating an agent rollback the same as a code rollback and forgetting to pin the model version. Teams frequently roll back their orchestration code after an incident, declare the issue resolved, and are surprised days later when the same behavior reappears because the underlying model provider silently updated the model the agent calls, and the rollback never addressed that dependency.
Cost-per-agent-run tracking
Token costs for agentic workflows can vary by an order of magnitude depending on how many reasoning steps, tool calls, and retries a given task requires, which makes aggregate monthly spend a poor signal for catching problems early. Mature agent operations teams track cost per completed run at the individual agent level, broken down further by task type where volume justifies it, so that a cost spike in one specific workflow does not get lost in an overall spending total that still looks reasonable.
This tracking discipline typically feeds two different audiences. Engineering uses it to catch runaway loops, excessive retries, or an agent falling back to an unnecessarily expensive model for a task a cheaper one could handle. Finance uses the same data, aggregated differently, to build the kind of full-picture total cost of ownership figures that a CFO or Chief AI Officer needs for board reporting, since sticker-price licensing costs typically represent only a fraction of what running an agent fleet actually costs once compute, monitoring, and human oversight are included.
| Cost signal | What it typically indicates |
|---|---|
| Rising cost per run with flat output quality | Model or prompt drift causing inefficient reasoning paths or unnecessary retries |
| Cost concentrated in a small number of runs | Edge cases triggering runaway loops or excessive tool calls |
| Cost falling alongside falling quality | Silent fallback to a cheaper model without a corresponding evaluation check |
| Stable cost per run, rising total volume | Healthy scaling; the agent is simply being used more, not misbehaving |
Decommissioning agents safely
Agents accumulate the same way unused software licenses and forgotten cron jobs do, except the risk profile is higher because a stale agent may still hold live credentials and permissions to take action in production systems. A formal decommissioning procedure is one of the most frequently skipped parts of agent operations, largely because there is rarely a dramatic moment that forces the issue the way a security incident forces a credential rotation.
A defensible decommissioning process includes a few consistent steps: confirming the business process the agent supported is genuinely retired or migrated to a successor agent, revoking the agent’s credentials and tool access before deleting its configuration, archiving its run logs and decision traces for the retention period required by relevant compliance rules, and removing it from the active agent registry so it no longer counts against the fleet’s monitoring and cost dashboards. Skipping the credential revocation step in particular has caused real incidents where a decommissioned agent’s leftover access was exploited or accidentally reused by a different system months later.
Escalation paths when an agent behaves unexpectedly
Even with strong monitoring, some agent behavior only becomes visible when a human downstream notices something is wrong, whether that is a customer flagging an odd response or an internal employee noticing a report does not match reality. Agent operations needs a clear, well-publicized escalation path for this kind of human-reported issue, separate from the automated alerting pipeline, because the two failure modes surface different kinds of problems. Automated monitoring catches statistical drift and cost anomalies well; it is much weaker at catching a plausible-sounding but subtly wrong piece of reasoning that a domain expert would spot immediately.
The organizations that handle this well give every agent a visible, documented owner and a simple reporting channel, similar to how a well-run IT department publishes a clear way to report a broken system, rather than expecting employees to track down whichever engineer happens to remember building a given agent.
What worked
Teams that built a single, centralized agent registry, listing every production agent, its owner, its current version, its permissions, and its escalation contact, cut incident response time significantly because responders no longer had to spend the first twenty minutes of an incident simply figuring out who was responsible for the misbehaving agent.
Building the operational foundation before scaling the fleet
Companies that try to scale from a handful of pilot agents to dozens of production agents without first building the operational foundation described above tend to hit the same wall: nobody can say with confidence how many agents are actually running, what they are allowed to do, or what they cost. This is the agent equivalent of “server sprawl” from the early cloud computing era, and it is solved the same way it was solved then, through a central registry, standardized deployment tooling, and a dedicated team whose job is the platform itself rather than any single agent. This is exactly the role a dedicated ML platform team is built to play, and organizations that stand up this function before fleet size gets out of hand consistently report fewer surprise incidents than those that try to retrofit governance onto an already-sprawling agent estate.
| Operational maturity stage | Typical fleet size | Primary risk if operations lag |
|---|---|---|
| Pilot | 1 to 3 agents | Low risk; issues are usually caught manually by the small team involved |
| Early scale | 4 to 15 agents | Registry and ownership gaps start to appear; incidents take longer to triage |
| Fleet operations | 16 to 50 agents | Without centralized tooling, cost and incident tracking become unreliable |
| Mature platform | 50-plus agents | Requires dedicated on-call rotation, versioned deployment, and formal decommissioning |
Frequently overlooked details in agent operations
- Silent model updatesAgent behavior can shift even when nobody on the team changed anything, because the underlying model provider updated the model version being called.
- Credential sprawlDecommissioned agents that keep live credentials and tool access are a common and underappreciated security exposure.
- Retry loops hiding in cost dataA small number of edge-case runs can quietly dominate total spend if retries and tool calls are not tracked per run.
- Ownership gapsAgents built quickly during a pilot phase often outlive the original builder’s involvement, leaving no clear owner when something goes wrong.
- Evaluation dataset stalenessA golden dataset used to catch drift needs periodic refreshing, or it stops reflecting the tasks the agent actually handles.
- Escalation channel awarenessEven a well-designed escalation path fails if the employees who would notice a problem do not know it exists.
Glossary
- AgentOps
- The operational discipline of deploying, monitoring, and maintaining AI agents in production, covering incident response, versioned deployment, cost tracking, and decommissioning.
- Canary release
- A deployment pattern where a new agent version handles a small share of real traffic while the previous version continues handling the rest, allowing comparison before a full rollout.
- Golden dataset
- A curated set of tasks with known-correct answers used to benchmark an agent’s accuracy and detect drift over time.
- Agent registry
- A centralized inventory of every production agent, including its owner, current version, permissions, and escalation contact.
- Model drift
- A gradual or sudden change in an agent’s output quality or behavior caused by changes to the underlying model, data, or environment rather than the agent’s own code.
Key Takeaways
- Agent operations is the SRE-style discipline of keeping production AI agents running safely, distinct from KPI design for measuring agent performance.
- On-call rotations for agents need domain expertise paired with engineering skill, since drifted output often looks superficially normal.
- Deployment and rollback must version both orchestration code and the underlying model or prompt configuration together.
- Cost-per-run tracking at the individual agent level catches problems that aggregate monthly spend figures hide.
- Decommissioning agents requires formally revoking credentials and access, not just deleting a configuration file.
- A visible, documented escalation path for human-reported issues catches problems automated monitoring misses.
- Building a central agent registry before scaling the fleet prevents the sprawl that makes later governance far harder.
FAQs
What is the difference between agent operations and agent performance KPIs?
Agent operations covers the runbook mechanics of keeping agents running safely in production: incident response, deployment, cost tracking, and decommissioning. Agent performance KPIs measure whether an agent is doing good work relative to a human baseline, a distinct and separate discipline covered in dedicated KPI design guides.
Who should be on call for agent incidents?
Effective rotations pair an engineer who understands the technical infrastructure with someone who understands the business logic the agent automates, since diagnosing subtly wrong agent output often requires domain knowledge a pure infrastructure engineer would not have.
Why can’t we just roll back an agent’s code like normal software?
Agent behavior depends on both the orchestration code and the underlying model or prompt version. A rollback that only reverts code will not fix a regression caused by an upstream model update, so both must be versioned and rolled back together.
How should we track the cost of running AI agents?
Track cost per completed run at the individual agent level, broken down by task type where volume justifies it, rather than relying on aggregate monthly spend, which can hide a small number of expensive edge-case runs dominating the total.
What does safely decommissioning an agent actually involve?
It involves confirming the supported process is truly retired, revoking the agent’s credentials and tool access before deleting its configuration, archiving its logs for the required retention period, and removing it from the active agent registry.
How is model drift different from a normal software bug?
Model drift can occur without any change to the agent’s own code, caused instead by the underlying model provider updating the model version being called or by gradual shifts in the data the agent encounters, making it harder to catch with traditional bug-tracking methods.
When should a company build a dedicated agent operations function?
Most companies start needing dedicated tooling and on-call processes once they move beyond a handful of pilot agents into double-digit fleet sizes, where informal, ad hoc tracking of ownership and cost stops being reliable.
What is a canary release for an AI agent?
It is a deployment pattern where a new agent version handles a small, monitored share of real traffic while the previous version continues serving the majority, allowing teams to compare output quality and cost before fully cutting over to the new version.
References
- UiPath, “AgentOps and Operationalizing AI Agents for the Enterprise”
- AIMultiple, “15 AI Agent Observability Tools in 2026: AgentOps and Langfuse”
- nasscom, “AgentOps: What It Takes to Run AI Agents in Production”
- Machine Learning Mastery, “The Practitioner’s Guide to AgentOps”
- Dysnix, “What Is AgentOps and How It Works”
- Barchart, “Belitsoft Report 2026: AI Agent Trends, Enterprises Run 12 AI Agents on Average”
For related coverage in this cluster, see how AI adoption is creating new roles like the ML platform engineer that agent operations depends on, how the Chief AI Officer uses the cost and incident data this discipline produces, our guide to the agent marketplace economy for how third-party agents change this picture, and our coverage of agentic ERP for a concrete example of agents running core business processes.
