What is AI observability?

Summary slides

Why traditional observability isn’t enough on its own

Traditional software observability answers a well-defined set of questions: is the service up, how fast is it responding, where are errors occurring in the code, and what’s the current load on the system. These questions matter just as much for an AI-powered system as for any other piece of software, and the tools built to answer them — metrics dashboards, structured logs, distributed tracing — remain useful. But they were designed around an assumption that AI systems break: that a request either succeeds or throws a clearly identifiable error, and that “success” is unambiguous once a response is returned.

An AI system can return a response that is, by every traditional metric, a complete success — no error thrown, reasonable latency, a well-formed output — while still being wrong, misleading, or unhelpful in a way that no traditional monitoring dashboard would ever flag. This is the gap AI observability exists to close: visibility into whether the actual content of what a system produced was good, not just whether the mechanical process of producing it completed without a technical failure. Closing that gap requires capturing information traditional observability tooling was never designed to capture — the actual retrieved context behind a generated answer, the reasoning steps an agent took, a confidence or faithfulness signal about whether an output was grounded in information — layered on top of not instead of the traditional metrics and traces that still matter for the system’s basic operational health and that no team should abandon just because a newer, AI-concern has come along.

Extending the three pillars: metrics, logs, and traces for AI

Traditional observability practice is often organized around three complementary types of data, and understanding how each one needs to be extended for an AI system is a useful way to structure what needs to be instrumented. Metrics are aggregated numerical measurements over time — request count, error rate, latency percentiles — and for an AI system, this category needs to expand to include quality-numbers: the rate at which retrieval returns low-confidence results, the rate at which a system falls back to a degraded response, the distribution of token counts and cost per request, the rate at which generated answers include a citation versus don’t. These are the AI-equivalents of the traditional metrics dashboard, and without them, a team is flying blind on exactly the dimension most likely to affect user trust.

Logs are structured records of discrete events, and for an AI system, this means capturing not just “request received, response sent” but the actual intermediate content: what query was sent to a retrieval system, what documents came back and with what relevance scores, what prompt was assembled and sent to the model, what the model’s raw response was before any post-processing. Traces connect a request’s full journey through a distributed system, and for an AI pipeline specifically, a trace needs to capture the sequence of retrieval, ranking, generation, and any tool calls as a connected whole, so that a slow or wrong final answer can be traced back to exactly which stage in that pipeline caused it, rather than being visible only as an undifferentiated total latency number or a single opaque “response was wrong” data point with nothing underneath it to explain why, which is the state most teams find themselves in before they’ve deliberately built this kind of tracing in.

Instrumenting the retrieval stage

Given that a generated answer is only as good as the material it was generated from, the retrieval stage of any RAG-style pipeline is one of the highest-value places to instrument carefully, because a retrieval-stage problem that goes unobserved will consistently masquerade as a generation-stage problem to anyone only looking at final outputs. Useful signals here include the actual query that was sent to the retrieval system — which is often not the user’s literal question, if any query rewriting happened first — the documents or chunks that were returned, their relevance or similarity scores, and whether any filtering or reranking changed that initial result set before it reached the model.

Capturing this data makes a common class of problem diagnosable that would otherwise be nearly invisible: a case where retrieval technically returned results, none of which were relevant, and the model did its best with irrelevant context rather than the honest “I don’t have this information” response a well-designed system should have produced instead. Without retrieval-stage instrumentation, this shows up only as “the answer was wrong,” with nothing to explain why — the model’s output looks like the obvious place to investigate, and a team can spend effort tuning prompts or trying a different model before ever discovering that the actual problem was upstream, sitting in what got retrieved in the first place, a full pipeline stage away from where the investigation started. This pattern — technically successful retrieval that was substantively useless — is one of the single most common root causes behind a wrong answer that otherwise looks, from the outside, exactly like a pure generation failure.

Instrumenting the generation stage

Once retrieval’s contribution is visible, the generation stage needs its dedicated instrumentation, distinct from simply logging the final output text, because the final text alone doesn’t reveal how that text was produced or how much it can be trusted. The full prompt sent to the model — not just the user’s original question, but the complete assembled context including any retrieved material, system instructions, and conversation history — is essential to capture, because subtle prompt construction bugs are a common source of quality problems that are completely invisible if only the final output is logged.

Model-level metadata matters too: which model version handled the request, what generation parameters were used, and, where the underlying provider exposes it, any confidence or log-probability signal about how certain the model was in producing that particular output. Faithfulness — whether the generated answer stays true to the retrieved context it was given, rather than drifting into unsupported claims — is harder to capture automatically and often requires a separate evaluation step, sometimes using a second model to judge the first model’s output against the source material it was given, but it’s one of the single most valuable observability signals an AI system can have, because it’s directly measuring the failure mode — confident, plausible-sounding, ungrounded output — that traditional monitoring is structurally blind to.

Instrumenting agentic and multi-step pipelines

For systems that go beyond a single retrieval-and-generate pass into multi-step agentic behavior, observability needs to extend further still because the number of individual decision points that can each independently go wrong grows substantially, and a single aggregated “task succeeded or failed” signal hides almost everything useful about where and why a failure happened. Every step in an agent’s reasoning loop is worth capturing as its discrete, structured event: what action the agent decided to take and why, what tool was called and with what parameters, what the tool returned, and how that result fed into the next decision.

This step-by-step trace is what makes an agent’s behavior debuggable rather than an opaque black box that either produces the right answer or doesn’t, with no visibility into which of its many internal decisions went wrong when it didn’t. It also enables a valuable kind of aggregate analysis that isn’t possible without it: looking across many traces to find patterns — a particular tool that gets called with malformed parameters unusually often, a type of task that consistently takes more steps than expected, a recurring point in a multi-step process where the agent’s reasoning tends to go off track. None of these patterns are visible from an aggregate success rate alone; they only become visible once individual step-level traces are captured systematically enough to be analyzed in bulk across many runs, which is exactly the kind of analysis that turns an occasional debugging session into an ongoing quality-improvement practice.

Cost and token observability

Beyond quality, AI systems introduce a cost dimension that traditional software observability rarely needs to track with the same granularity, because unlike a typical web request, the cost of an individual AI request can vary enormously based on how much context was retrieved, how long the generated response was, and how many steps an agentic task took — which makes cost itself a signal worth observing carefully, not just a number that shows up on a monthly invoice with no way to trace which requests drove it.

Token-level tracking — how many tokens were consumed on the input side, the output side, and across any intermediate steps in a multi-call pipeline — gives visibility into where cost is concentrated, which matters because it’s rarely evenly distributed: a small number of unusually complex requests, or a task type that consistently requires more retrieval or more agentic steps than others, often accounts for a disproportionate share of total cost, in much the same way a small fraction of database queries can dominate a traditional system’s compute bill. Without this granularity, cost management tends to happen reactively, in response to a surprising invoice, rather than proactively, by identifying and addressing the request patterns driving spend before they become a large line item. Tracking cost per request type, per user, or per feature — wherever the system has meaningful segments worth distinguishing — turns an otherwise opaque aggregate number into something a team can act on, whether that means optimizing an expensive pipeline stage or reconsidering whether a particular feature’s cost is proportionate to the value it provides.

Latency observability across a multi-stage pipeline

Latency matters for AI systems in the same basic sense it matters for any software, but the way to observe it usefully is different, because a typical AI request isn’t a single operation — it’s a chain of stages, and the total latency a user experiences is the sum of however many of those stages a request needed. A single aggregate “response time” metric, the kind that’s often sufficient for a simpler service, hides exactly the information needed to improve latency for an AI pipeline, because it doesn’t reveal whether a slow response was slow because retrieval took a long time, because the model took a long time to generate, because a tool call was slow, or because several of those stages compounded together.

Stage-level latency instrumentation — timing retrieval, generation, and any tool calls separately, within the same overall trace — makes it possible to identify where latency is coming from and target improvement effort at the stage that’s the bottleneck, rather than guessing. It also reveals a pattern that’s easy to miss with only an aggregate number: latency in AI pipelines is often much more variable than in traditional services, because certain request types require more retrieval or more generation than others, and a percentile-based view — the typical latency versus the slower tail — tends to be considerably more informative than a single average, since the average can look perfectly reasonable even while a meaningful fraction of requests are experiencing a much worse, and much more noticeable, experience than that average would suggest — and it’s precisely that slower tail, not the comfortable average, that tends to generate the visible user complaints a team eventually has to respond to anyway. Stage-level percentile tracking makes it possible to see which stage is responsible for that slow tail, rather than only knowing that a tail exists somewhere in an undifferentiated total.

Capturing human feedback as a first-class observability signal

Everything covered so far describes signals a system can generate about itself — retrieval scores, faithfulness checks, latency and cost breakdowns — but there’s a second, equally important category of observability data that comes from the people using the system, and it’s worth treating with the same deliberateness as any automated signal rather than as an afterthought bolted on separately. Explicit feedback — a thumbs up or down, a rating, a flag that an answer was wrong — is the most direct version of this and capturing it in a way that’s linked back to the full trace of the request it’s responding to is what makes it useful rather than just a satisfaction percentage with nothing underneath it: a downvoted response becomes actionable the moment it can be traced back to exactly what was retrieved and generated in that case, rather than sitting as an isolated, context-free data point.

Implicit feedback matters just as much, and is available far more often than explicit ratings ever will be, since most users never bother to click a thumbs-down button even when an answer disappointed them. A user immediately rephrasing and re-asking a very similar question is a strong signal that the first answer didn’t land. A user abandoning a conversation right after receiving a response, or escalating to a human agent shortly after an AI-generated answer, are both meaningful proxies for dissatisfaction that don’t depend on anyone explicitly reporting it. None of these implicit signals are as clean or as unambiguous as an explicit rating, and each one individually carries some noise — a rephrased question doesn’t always mean the first answer was wrong, sometimes a user is just exploring a topic further — but in aggregate, across enough interactions, these implicit signals tend to correlate meaningfully with quality problems, and a team that only watches explicit feedback is working from a small, self-selected sample while ignoring a much larger, freely available signal sitting in ordinary usage patterns that were already being recorded anyway.

What to store, for how long, and the privacy trade-offs involved

Capturing this much detail — full prompts, retrieved content, intermediate reasoning steps, user feedback — inevitably raises a question that traditional infrastructure observability rarely has to grapple with as seriously: how much of this, especially when it includes user input and potentially sensitive retrieved content, should be stored, for how long, and who should be able to see it. This isn’t a question to leave unaddressed until it becomes a compliance problem; it needs to be part of the observability design itself, alongside the technical instrumentation decisions covered throughout this page.

A reasonable starting approach is to distinguish between what needs to be retained in full detail, what can be safely aggregated or anonymized, and what shouldn’t be retained past the point where it’s no longer operationally useful. Full request and response content is often necessary for debugging and quality evaluation, but it may need to be access-controlled more tightly than aggregate metrics, redacted of certain categories of sensitive information before storage, or retained for a shorter window than less sensitive operational data like latency and cost metrics, which typically carry much lower privacy risk and can reasonably be kept much longer for trend analysis. Different systems will land in different places on these trade-offs depending on what kind of data they handle — a system processing sensitive personal or medical information needs meaningfully stricter handling than one answering general product questions — but the underlying principle holds everywhere: observability instrumentation needs to be designed with an explicit answer to what gets stored and for how long, rather than defaulting to “log everything, forever,” which tends to accumulate privacy and compliance risk quietly, in the background, well before anyone notices it’s become a problem worth addressing directly.

Detecting drift before it becomes a visible problem

Everything covered so far describes observability at the level of an individual request, but AI systems also need observability at a longer time horizon, because their behavior can change gradually, without any single incident marking the moment it happened — a pattern usually called drift. A knowledge base gets updated in ways that shift what retrieval surfaces, an underlying model provider updates their model with subtly different behavior, or the mix of questions users ask shifts over time as a product or user base evolves, and any of these can degrade a system’s real-world quality steadily enough that no single day looks alarming, even as the trend over weeks or months clearly is.

Observing for drift specifically means tracking the same quality and confidence signals discussed throughout this page not just as point-in-time numbers, but as trends over time, with alerting tuned to catch a gradual decline rather than only a sudden spike. It also means periodically re-running a fixed, stable set of test cases against the live system — the same representative questions, asked the same way, at regular intervals — specifically to detect the case where the system’s behavior on a fixed, unchanging input has shifted over time, since that’s a signal that something in the environment around the system changed, even when nothing in the system’s code did — a distinction ordinary, traditional code-based regression testing was simply never built to catch. Without this kind of longitudinal observability, drift tends to be discovered the same way it’s discovered without any monitoring at all — by a user complaint, well after the decline has already meaningfully affected people relying on the system, at a point considerably later than a team would have wanted to know.

Building dashboards and alerting that reflect AI-risk

All of this instrumentation only produces value if it’s surfaced somewhere a team looks regularly and reacts to, which means dashboard and alerting design deserves its deliberate attention rather than being an afterthought layered on top of whatever data happens to get logged. A dashboard built around only traditional infrastructure metrics — uptime, latency, error rate — will look reassuringly healthy even while the AI-signals discussed throughout this page are quietly deteriorating, which is exactly the blind spot this whole discipline exists to close, and a dashboard that doesn’t surface those AI-signals prominently has essentially recreated that blind spot in dashboard form, just with a more sophisticated-looking set of charts sitting on top of it.

A well-designed AI observability dashboard puts quality signals — faithfulness scores, fallback rates, low-confidence retrieval rates — alongside the traditional operational metrics, not in a separate, easy-to-ignore tab that only gets checked occasionally. Alerting thresholds need to be calibrated specifically for these AI signals too, which usually means thinking in terms of gradual trend changes rather than only hard, sudden threshold breaches — a fallback rate that’s crept up by a meaningful amount over the past week deserves attention even if it hasn’t crossed some dramatic absolute threshold, because that kind of gradual creep is exactly the drift pattern discussed above, and catching it early, while it’s still a minor trend, is considerably more useful than only catching it once it’s become large enough to breach a hard-coded alert threshold that was really only ever designed to catch sudden, dramatic failures rather than slow, cumulative ones.

How observability relates to evaluation, and why both are needed

With this much instrumentation in place, it’s worth being precise about a distinction that’s easy to blur: observability and evaluation are related but different activities, and a team that only builds one of them ends up with a gap. Evaluation is the practice of deliberately testing a system against a curated set of representative scenarios, usually before a change ships, to answer the question “is this version of the system good enough.” Observability is the practice of watching a system’s actual behavior on live traffic, continuously, to answer the question “is the system behaving well right now, on whatever users are sending it.”

A system can score well on a curated evaluation set and still perform poorly on traffic, because traffic inevitably includes request patterns, phrasings, and edge cases the evaluation set’s authors didn’t think to include — which is precisely why observability matters even for a system that passed evaluation with strong marks. Conversely, observability alone, without a deliberate evaluation practice run before changes ship, means every quality regression is discovered only after it’s already reached users, rather than caught in a controlled test before release. The two practices feed each other in a useful way: patterns discovered through observability — a recurring failure mode, a category of question the system handles poorly — are exactly the kind of finding that should get folded back into the evaluation set, so that the next version of the system is deliberately tested against the real-world failure that observability surfaced, rather than the evaluation set staying frozen at whatever scenarios its authors happened to think of when it was first written, months or years before the system encountered the patterns evaluation never anticipated.

Observability needs across development, staging, and production

The level and kind of observability that’s useful also shifts across the different environments a system passes through on its way to being used by people, and treating every environment identically tends to under-serve at least one of them. During active development, the most valuable observability is often the most granular and immediate — full traces available instantly for the exact request a developer just ran, so they can inspect precisely what happened in that one case while iterating quickly on a prompt or a retrieval configuration. Heavy aggregation and long-term trend dashboards matter much less here, because the developer already knows exactly which request they care about and just needs to see everything about it.

In staging, observability starts to shift toward validating behavior across a broader, more representative range of scenarios before a change ships to users — this is where evaluation and observability overlap most directly, running a fuller battery of test cases and inspecting their traces in aggregate to catch regressions before they reach production traffic at all. In production, observability needs to operate at scale, continuously, with the aggregate metrics, trend detection, and alerting discussed throughout this page doing most of the work, since no team can manually inspect every individual production trace the way a developer inspects a handful of requests during active local development. A mature observability setup supports all three of these modes without forcing a team to choose only one, because each environment needs a different balance of depth-per-request versus breadth-across-requests, and building for only one mode — usually production-scale aggregation, since that’s the mode most existing tooling is built around — tends to leave developers under-served during the exact iterative, fast-feedback work where detailed, immediate traces matter most.

Using observability data to debug a problem

The test of whether an observability setup is working is whether it helps when something goes wrong, so consider a concrete scenario: users start reporting that a support assistant’s answers about a product feature have gotten noticeably worse over the past week. Without proper instrumentation, this report is nearly impossible to act on directly — the team can try to reproduce the problem by asking similar questions themselves, but that’s slow, unreliable, and gives no guarantee of finding the same failure a user hit. With proper observability in place, the investigation looks entirely different: filter traces to that topic over the relevant time window, and look first at retrieval — did the documents retrieved for these questions change, or degrade in relevance, around the time the complaints started, perhaps because of a recent documentation update.

If retrieval looks unchanged and reasonable, the next place to look is generation — has faithfulness scoring on these requests dropped, suggesting the model started drifting from the retrieved context rather than a retrieval problem at all. If neither retrieval nor generation shows an obvious change, cost and latency traces for these requests are worth checking too, since a timeout or a truncated context under unusual load can degrade quality in ways that look, from the outside, identical to a retrieval or generation problem, even though the underlying cause is really an infrastructure issue rather than a content issue at all. This systematic, trace-by-trace, stage-by-stage investigation — moving methodically through retrieval, then generation, then infrastructure, checking each against what changed and when — is only possible because each of those stages was instrumented separately in the first place; without that instrumentation, the same investigation collapses into guesswork, trying prompt tweaks and configuration changes somewhat at random and hoping one of them happens to fix a problem nobody can see clearly enough to diagnose with any confidence.

Once the actual cause is identified through this process — say, it turns out to be a documentation update that inadvertently changed how a feature’s content was chunked, degrading retrieval relevance for questions about it — the same observability data closes the loop in a second, equally important way: it lets the team verify the fix worked, by watching the same faithfulness and retrieval-confidence signals for that topic recover after the fix ships, rather than simply assuming a plausible-sounding fix worked and moving on without confirmation. This closing-the-loop step matters more than it might seem, because it’s exactly what distinguishes a resolved incident from one that merely stopped generating complaints for a while, only to resurface later in a slightly different form because the actual underlying cause was never confirmed to be fixed at all — just addressed with a change that seemed reasonable and happened to coincide with the complaints temporarily dying down.

Common mistakes teams make with AI observability

A consistent set of mistakes shows up across teams building observability for AI systems, worth naming directly because each one tends to recreate exactly the blind spot this whole discipline exists to eliminate. The most common is instrumenting only the traditional operational metrics and treating that as sufficient, which leaves a team with a dashboard that looks reassuringly healthy while the actual quality of what the system produces silently degrades, invisible to every metric being watched. A second is capturing final outputs without capturing the intermediate stages that produced them — logging what a system said without logging what it retrieved or what prompt was assembled — which makes every investigation into a wrong answer start from close to zero, with no trail to follow back to an actual cause.

A third is building quality signals like faithfulness scoring only as an occasional, manual evaluation exercise rather than as continuous, automated instrumentation running on live production traffic, which means quality problems are caught only when someone happens to go looking for them rather than surfaced automatically the way a traditional error rate would be. A fourth is treating cost and latency purely as aggregate numbers rather than breaking them down by pipeline stage and request type, which leaves a team knowing that cost or latency is a problem without any actionable path to knowing specifically where to focus effort to improve it. And a fifth, tying back to drift discussed earlier, is building observability only for point-in-time debugging and never for longitudinal trend detection, which means a system can be silently getting worse for weeks before anyone notices, simply because nobody built the kind of ongoing, trend-aware monitoring that would have caught the decline while it was still small and easy to address, rather than after it had already become large enough to generate a wave of user complaints that no dashboard was ever designed to prevent in the first place.

What connects every one of these mistakes, and every instrumentation decision discussed throughout this page, is the same underlying reframing: for an AI system, “working” and “producing a technically valid response” are not the same claim, and observability is the discipline of being able to tell the two apart on an ongoing basis rather than assuming that success at the infrastructure level implies success at the level that matters to the people relying on the system. A system instrumented only for traditional operational health will always look reassuring on its own dashboard, right up until a user complaint reveals a gap between what the dashboard showed and what was happening — and by then, the gap has usually existed, quietly, for considerably longer than anyone realized. Building observability that closes that gap from the start, across retrieval, generation, agentic steps, cost, latency, human feedback, and long-term drift alike, is what turns an AI system from something a team hopes is working into something a team can demonstrate, with evidence, is working — and can act on quickly and specifically the moment that evidence says otherwise.