LLM application observability tracks what classical APM and agent tracing do not: per-request cost in dollars, a latency waterfall across every model and retrieval call, and quality signals like LLM-as-judge scores and hallucination rate. It applies to any LLM-powered app, chatbot, RAG pipeline, or summarization service, whether or not that app is agentic.
| Dimension | LLM application observability (this piece) | Agent observability | Model monitoring |
|---|---|---|---|
| Unit of analysis | Per-request trace across the LLM call chain | Full autonomous run with tool calls | Feature and prediction distributions over a serving window |
| Core question | What did this request cost, how long did each hop take, and was the answer good? | What did the agent do, and can I replay it? | Has the input or output distribution shifted from training? |
| Cost attribution | Central: tokens times price, per model and per call | Secondary, if covered at all | Not applicable |
| Quality scoring | LLM-as-judge, hallucination rate, faithfulness, user feedback | Task success and tool-selection correctness, not text quality | Accuracy, precision, recall against delayed ground truth |
| Typical tools | Langfuse, Helicone, LangSmith, Arize | Agent trace stores, replay harnesses | Evidently AI, WhyLabs, Fiddler, Arize |
| Applies to agents? | Yes, whether or not the application is agentic | Agent-specific by definition | Not LLM-specific; classical supervised models |
Three separate posts on this site sit near each other and get confused constantly, so it is worth saying plainly where each one ends. For agent-specific replay and tool-call tracing, see Agent Observability: Tracing, Replay, and Debugging Autonomous Workflows. For classical ML model drift and skew monitoring, see Model Monitoring in Production: Drift, Skew, and Silent Failure. This piece covers the LLM-application observability layer: cost, latency, and quality, which applies whether or not the application is agentic. A single-shot summarization endpoint that never calls a tool still needs all three; an autonomous agent needs all three plus everything in the agent observability piece.
Why does an LLM application need its own observability layer?
Because none of the three questions that matter for an LLM-backed feature are answered by conventional application performance monitoring. A web server dashboard tells you request rate, error rate, and p99 latency. None of that tells you which model call inside a request cost four cents versus forty, which stage of a five-step pipeline added the two extra seconds a user noticed, or whether the answer the system produced was actually true.
Traditional APM was built for a world where cost per request was effectively zero and correctness was binary: the code either threw an exception or it did not. LLM applications invert both assumptions. Cost varies by an order of magnitude between a cache hit and a cache miss, between a small model and a large one, between a three-turn conversation and a thirty-turn one. And correctness is not binary at all. A response can be fluent, well-formatted, confidently delivered, and still wrong, with no exception anywhere in the stack to flag it.
The three pillars: cost, latency, and quality
Every mature LLM observability setup, regardless of vendor, converges on the same three pillars. They are distinct enough that a team can be excellent at one and blind in another, which is exactly the failure mode this section is meant to prevent.
Per-request cost attribution
The mechanics are simple: multiply input tokens and output tokens by the provider’s per-token price for the model actually used, then sum across every call a single request triggers. The difficulty is never the arithmetic. It is that a single user-facing request in a RAG or multi-step pipeline often fans out into several distinct model calls, an embedding call for the query, a rerank call, a generation call, and sometimes a follow-up call for citation formatting or a safety check, each potentially on a different model with a different price per million tokens.
Without per-call attribution, a team only sees the blended total on the monthly invoice. With it, a team can answer the question that actually drives cost decisions: which specific call in which specific feature is spending the money, and would a cheaper model in that one slot change the user-visible answer at all.
| Call in pipeline | Model | Input tokens | Output tokens | Approx. cost per request | Share of total cost |
|---|---|---|---|---|---|
| Query embedding | Small embedding model | 40 | 0 | Under $0.0001 | Under 1% |
| Reranking | Cross-encoder reranker | 1,200 | 0 | $0.0003 | 2% |
| Answer generation | Frontier chat model | 3,500 | 400 | $0.014 | 82% |
| Citation formatting pass | Small chat model | 500 | 150 | $0.002 | 12% |
| Safety classification | Small classifier model | 400 | 10 | $0.0002 | 1% |
This is an illustrative, rounded breakdown, but the shape is representative of what teams actually find once they attribute cost per call rather than per request: generation dominates, and the cheapest-looking optimization, swapping the small formatting or classification calls, barely moves the total. The finding that reliably does move the total is reducing generation input tokens through better retrieval, or reusing a cached prefix so the bulk of that 3,500-token input never gets billed again. Reordering a prompt so static content precedes variable content, covered in depth in Prompt Caching and Batching: Cutting Inference Costs Without Losing Quality, is consistently the highest-leverage change available once per-call attribution reveals where the tokens actually go.
Attribution also needs dimensions beyond the call itself. Tagging every trace with a user ID, a feature name, and a session ID turns a flat cost number into an answerable set of business questions: which customer segment is expensive to serve, which feature should be re-priced or rate-limited, and whether a recent prompt change quietly doubled the input token count for every request. The organizational habit of treating unit economics as a first-class operational concern, not just an LLM concern, is covered more broadly in our piece on observability cost control.
Latency waterfall across the call chain
A single “response time” number hides more than it reveals once a request involves more than one hop. A RAG pipeline answering one question typically involves retrieval, reranking, and generation as at least three sequential stages, and each has a distinct latency profile with a distinct fix.
Figure 1: A latency waterfall for a single RAG request
A horizontal bar chart stacking five segments left to right for one traced request: query understanding and embedding (roughly 80 milliseconds), hybrid retrieval against the vector and keyword indexes (roughly 150 milliseconds), cross-encoder reranking of the candidate set (roughly 300 milliseconds, frequently the largest non-generation segment), prompt assembly and network overhead (roughly 60 milliseconds), and generation up to the final token (900 milliseconds to several seconds depending on output length). The waterfall view, rather than a single total, is what lets an engineer see that reranking, not retrieval, is the segment worth optimizing first.
The reason a waterfall view matters more than a single latency number is that each stage has a completely different lever. Slow retrieval usually means an index problem or a metadata filter running too late. Slow reranking usually means too large a candidate set being scored by too large a cross-encoder. Slow generation usually means output length, a queued or rate-limited provider, or a model tier heavier than the task requires. Without the breakdown, every latency complaint gets diagnosed by intuition, and the wrong stage gets optimized. The general discipline of decomposing a distributed request into per-hop spans long predates LLM applications; the practice is described in detail in our guide to distributed tracing for microservices debugging, and the same span-per-hop structure is exactly what tools like Langfuse and LangSmith apply to an LLM call chain specifically. The architecture of the pipeline being traced, including why reranking is so often the largest addressable segment, is covered in RAG Architecture in 2027: Hybrid Retrieval, Rerankers, and Graph Indexes.
Time to first token deserves separate tracking from total completion time. A user staring at a blank screen for 900 milliseconds before any text appears experiences the wait very differently than a user watching text stream in from millisecond 200 onward, even if the two requests finish at the same total time. Streaming-aware latency dashboards separate these two numbers deliberately rather than reporting one blended figure.
Quality observability: LLM-as-judge and hallucination rate
Cost and latency are numeric and unambiguous. Quality is neither, and that is exactly why it is the pillar most often skipped, even though it is the one users actually feel. A response can be fast and cheap and still be wrong, unhelpful, or fabricated, and none of the infrastructure metrics discussed so far will show it.
Two production-shaped signals dominate real quality observability setups. The first is LLM-as-judge scoring, running a separate model against each production output, or a sample of it, using a rubric that grades relevance, coherence, or adherence to instructions. The second is hallucination and faithfulness rate, specifically for retrieval-grounded applications, measuring whether a generated claim is actually supported by the retrieved context it was supposed to be grounded in. Arize’s open-source Phoenix evaluators, for example, report roughly 82 percent agreement with human labels on hallucination detection with a false-positive rate near 12 percent on their published benchmark set, numbers worth knowing because they set a realistic ceiling: an automated judge is a calibrated instrument, not a certainty machine.
Both signals need to be tracked as trends over time and segmented by prompt version, model version, and feature, not read as a single global average. A hallucination rate that looks fine in aggregate can be hiding a segment, a document type, or a query pattern where it is far worse. The methodology for building a judge that is actually trustworthy, including the biases that quietly inflate scores, is covered fully in LLM-as-Judge: Setting Up Automated Grading Without Fooling Yourself, and the specific techniques for catching fabricated claims before a user acts on them are covered in Hallucination Detection: 10 Techniques That Actually Reduce False Claims. Wiring both of those into a repeatable suite that runs on every deploy, rather than as an occasional audit, is the subject of LLM Evaluation Frameworks: How to Build an Eval Suite That Catches Regressions.
User feedback, thumbs up or down, a regeneration click, a copy action, is the cheapest quality signal available and the most frequently under-instrumented. It will not catch subtle factual errors a user does not notice, but it correlates strongly with the failures users actually complain about, and it costs nothing beyond a button and a logging call.
Instrumenting the stack: traces, spans, and the OpenTelemetry GenAI conventions
The structural shape underneath all three pillars is the same: one trace per user-facing request, with a nested span per model or tool call inside it, and cost, latency, and quality attributes attached at the span level so they can be queried independently or rolled up together. OpenTelemetry’s GenAI semantic conventions, published under the gen_ai namespace, standardize exactly this: attributes for the model requested, input and output token counts, and finish reason, attached consistently regardless of which provider served the call. Adopting a shared naming convention rather than a bespoke schema per team is what makes cross-team dashboards and vendor migration possible later, a broader argument made in our piece on OpenTelemetry adoption.
One practical detail trips up almost every team building this for the first time: full prompt and completion text does not belong in indexed span attributes, which have size limits and are meant for filtering, not payload storage. The convention is to store token counts and model identifiers as attributes, and the actual prompt and response text as span events or in a separate, access-controlled trace store. Getting this ordering backward is how teams end up with either truncated prompts they cannot debug from, or an indexing system straining under multi-kilobyte attribute values it was never designed to hold.
The tool landscape: Langfuse, Helicone, LangSmith, and Arize
None of the major LLM observability platforms are pure supersets of each other; each optimizes for a different starting condition.
| Tool | Integration model | Cost tracking | Quality and eval features | Best fit |
|---|---|---|---|---|
| Langfuse | SDK instrumentation, OpenTelemetry-native, open-source self-host | Per-call token and cost attribution with custom pricing maps | Deep built-in eval primitives, prompt versioning, dataset-based regression testing | Teams wanting the most complete open-source picture across cost, latency, and quality |
| Helicone | Proxy in front of the model API, near-zero code change | Automatic, since every call already passes through it | Lighter-weight; strongest for cost and basic quality tagging | Small teams wanting a five-minute integration under moderate LLM spend |
| LangSmith | Native to LangChain and LangGraph, SDK elsewhere | Prebuilt cost dashboards with custom cost overrides | Strong trace and dataset tooling, tightest fit inside its own framework | Teams already standardized on LangChain or LangGraph |
| Arize | SDK or OpenTelemetry, plus the open-source Phoenix project | Supported, secondary to its evaluation focus | Widest library of research-backed evaluators, including hallucination and faithfulness scoring | Teams whose primary open question is quality, or who also run classical ML models |
The practical selection question is rarely which tool has the most features. It is which pillar is currently the weakest at your organization. A team drowning in an unexplained bill should weight cost tracking and per-call attribution first, which points toward Helicone’s low-friction proxy or Langfuse’s pricing maps. A team fielding complaints about wrong answers should weight eval depth first, which points toward Arize or Langfuse’s dataset-based regression tooling. A team already deep in LangChain gets the least friction from LangSmith simply because the framework and the observability layer were built by the same team.
Building dashboards and alerts that reflect all three pillars
A dashboard with only latency and error rate panels will look perfectly healthy on the day an LLM application quietly starts hallucinating more, because nothing about a fabricated but fluent answer trips a latency or error alert. The dashboards that catch real incidents put cost, latency, and quality side by side, sliced by the same dimensions: model version, prompt version, and feature.
Figure 2: A three-pillar dashboard layout
Three panels arranged in a single row, sharing a common time axis and the same filter controls for model version and feature. The left panel tracks cost per request and total daily spend by model. The center panel tracks the latency waterfall described earlier, broken into retrieval, rerank, and generation segments. The right panel tracks quality: LLM-as-judge score trend, hallucination rate trend, and thumbs-down rate, each plotted against the same prompt-version markers so a regression can be traced back to the exact deploy that caused it.
Alert thresholds need the same care given to any production system, and the general discipline of choosing thresholds that a team will actually act on, rather than ones that generate noise until everyone tunes them out, is covered in our piece on SLOs that teams actually use. For LLM applications specifically, a quality regression should page differently than a cost spike: a rising hallucination rate is a correctness problem affecting users right now, while a cost spike is a budget problem that rarely needs a 2 a.m. wakeup. Wiring both into a shared, rehearsed response process, rather than routing every alert to the same channel, is exactly the kind of preparation described in our guide to incident management.
Common mistake
Teams instrument cost and latency in the first sprint, ship, and never come back to add quality observability, because it is the pillar without an obvious infrastructure metric to bolt on. Months later a prompt or model change quietly degrades answer quality, and nothing in the dashboard moves, because nothing in the dashboard was measuring quality at all. The cost and latency panels stay green while user trust erodes underneath them. Quality instrumentation is not an optional fourth phase; it needs to ship in the same release as the feature it is measuring, even if the first version is only a thumbs-up and thumbs-down button and a weekly manual sample.
What worked
Teams that got the most value fastest did not build a custom cost, latency, and quality stack from scratch. They adopted the OpenTelemetry GenAI conventions from day one, even before choosing a vendor, so every span already carried standard token, model, and cost attributes. Switching from a proxy-based tool to a full tracing platform later, or running both side by side during a migration, required no re-instrumentation because the underlying data was already vendor-neutral. The teams that skipped this and instrumented directly against one vendor’s SDK paid for it later with a full re-instrumentation project the first time they needed to switch tools or add a second one for a different pillar.
Glossary
- Trace
- The complete record of one user-facing LLM request, containing every nested call it triggered.
- Span
- A single step within a trace, such as one model call, one retrieval, or one rerank pass, carrying its own timing and attributes.
- Token attribution
- Assigning input and output token counts, and the resulting cost, to the specific call and feature that produced them rather than to a blended total.
- LLM-as-judge
- Using a separate language model to grade production output against a rubric, standing in for human review at a fraction of the cost.
- Hallucination rate
- The measured proportion of generated claims that are not supported by the source material a system was meant to be grounded in.
- Faithfulness
- How closely a generated answer’s claims track the retrieved or supplied context, distinct from whether the answer merely sounds correct.
- Latency waterfall
- A per-stage breakdown of where time is spent across a multi-call pipeline, such as retrieval, reranking, and generation, rather than one total response time.
- GenAI semantic conventions
- The OpenTelemetry attribute standard under the gen_ai namespace for naming model, token, and cost fields consistently across providers and tools.
Key Takeaways
- LLM application observability covers cost, latency, and quality, a distinct layer from agent-specific replay tracing and from classical ML model drift monitoring.
- Per-request cost attribution requires multiplying tokens by price at the individual call level, since one user request often fans out into several model calls with different prices.
- A latency waterfall across retrieval, reranking, and generation reveals which stage to optimize; a single total response time hides that entirely.
- Reranking is frequently the largest addressable latency segment after generation itself, and prompt restructuring for caching is frequently the largest addressable cost lever.
- Quality observability, LLM-as-judge scoring and hallucination rate tracking, is the pillar most often skipped, yet the one users feel most directly.
- Adopting OpenTelemetry’s GenAI semantic conventions early keeps a team vendor-neutral and avoids a costly re-instrumentation project during a later tool migration.
- Langfuse, Helicone, LangSmith, and Arize each optimize for a different starting condition; the right choice depends on which of the three pillars is currently weakest.
FAQs
What is the difference between LLM application observability and agent observability?
LLM application observability measures cost, latency, and quality across any LLM-powered request, agentic or not. Agent observability specifically covers replay safety, tool-call capture, and redaction for multi-step autonomous runs. An agentic application needs both layers; a simple chatbot or summarizer only needs the first.
How is per-request LLM cost actually calculated?
Input and output token counts for each model call are multiplied by that model’s per-token price, then summed across every call one user request triggers. Because a single request can involve embedding, reranking, generation, and formatting calls on different models, attribution has to happen per call, not just once at the request level.
Why does a single latency number hide more than it reveals?
A RAG request typically involves retrieval, reranking, and generation as sequential stages, each with a different bottleneck and a different fix. A single total conflates a slow reranker with a slow generation call, leading teams to optimize the wrong stage entirely.
What does hallucination rate actually measure in production?
It measures the proportion of generated claims not supported by the retrieved or supplied source material. Automated evaluators like Arize’s Phoenix report around 82 percent agreement with human labels on published benchmarks, which sets a realistic ceiling rather than a guarantee of certainty.
Should quality observability be added after launch or before?
Before, or in the same release as the feature itself. Cost and latency dashboards can look completely healthy while a prompt or model change quietly degrades answer quality, since nothing in an infrastructure metric reflects a fluent but wrong answer.
Which LLM observability tool should a small team start with?
It depends on which pillar is weakest. Helicone’s proxy model gives near-zero-effort cost visibility for teams under moderate spend. Langfuse gives the most complete open-source picture across all three pillars. LangSmith fits best inside an existing LangChain or LangGraph stack.
Do the OpenTelemetry GenAI conventions replace the need for a dedicated LLM observability tool?
No. They standardize the attribute names and span structure so data stays vendor-neutral across providers and frameworks, but a dedicated tool still provides the trace storage, dashboards, cost pricing maps, and evaluation pipelines a team needs to actually act on that data day to day.
Why do teams under-instrument quality compared to cost and latency?
Cost and latency map cleanly to existing infrastructure metrics that engineers already know how to build dashboards for. Quality has no equivalent default metric, requires either a judge model or human labels, and its absence produces no error, only a slow, unnoticed decline in answer usefulness.
References
- OpenTelemetry. Inside the LLM Call: GenAI Observability with OpenTelemetry.
- Langfuse. Token and Cost Tracking documentation.
- LangChain. LangSmith cost tracking documentation.
- Arize AI. Phoenix: open-source LLM tracing and evaluation.
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
- Edge, D., et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130.