What is agent observability?

Summary slides

Why agent observability needs its dedicated practice

A conventional software system’s actual behavior is typically traceable through a relatively linear sequence of function calls and clear service boundaries, and general observability tooling built around that assumption works reasonably well for tracing a request through such a considerably more predictable system. An AI agent breaks this relatively linear assumption considerably, since its actual path through a task simply isn’t fixed in advance at all, it emerges dynamically as the agent reasons, calls tools, observes results, and adjusts its plan based on what it learns along the way, which means the trace of an agent trajectory looks considerably less like a straightforward, predictable call stack and considerably more like a branching, adaptive sequence of decisions that general-purpose observability tooling was never specifically built to represent well at all.

This structural difference is exactly why agent observability has to exist as its dedicated practice, with its data model for capturing plans, tool calls, and intermediate reasoning as first-class, structured elements rather than treating an agent’s entire trajectory as an undifferentiated block of log output. A team relying purely on general-purpose logging to understand agent behavior typically finds itself unable to answer the most basic, fundamental diagnostic question a failed trajectory raises, at which step did this go wrong and why, precisely because general-purpose logging wasn’t ever built with an agent’s branching structure in mind at all.

What a full agent trace captures step by step

A truly complete agent trace records the initial goal or task the agent was given, the agent’s plan for approaching that task, ideally captured explicitly, directly rather than only inferred well after the fact from its subsequent actions, and then a sequential, ordered record of every individual step the agent really took, each tool call made along with its exact parameters, each result that call returned, and any point where the agent’s plan visibly changed in direct response to what it learned from a result. This step-by-step, granular detail is precisely what makes a trace useful for diagnosis, since a trace that only ever records the agent’s final output, without this intermediate, granular detail, gives a team essentially nothing to work with when trying to understand why that final output turned out the way it did in the end.

Beyond the raw, ordered sequence of actual actions, a useful, well-built trace also captures timing information for each individual step, how long a tool call took, how much of the total trajectory’s time was spent on model reasoning versus waiting on external tool results, and token or cost information attributed specifically to each step rather than only to the trajectory as a whole. This additional detail lets a team diagnose not just correctness problems, where did the agent go wrong, but efficiency problems too, where specifically did the agent spend disproportionate time or cost relative to the actual value that step contributed to the overall, larger task.

How to visualize and navigate a long agent trajectory

A raw, unprocessed trace, however complete, quickly becomes unwieldy to practically read once a trajectory extends across dozens of individual steps, which is exactly why useful agent observability tooling presents a trace visually, as a structured sequence or tree a person can navigate, expanding any step to see its full, granular detail while collapsing the surrounding steps into a compact, condensed summary view, rather than forcing a reviewer to scroll endlessly through an undifferentiated wall of raw log text trying to manually, painstakingly reconstruct the trajectory’s actual shape by hand.

Effective, well-designed visualization also needs to surface the trajectory’s branching and backtracking structure explicitly and clearly, directly showing the exact point where an agent abandoned one approach and started an entirely different one instead, rather than presenting every single step as though it were merely part of one uniform, linear sequence when the agent’s actual behavior involved meaningful pivots along the actual way. A visualization that flattens this branching structure into a purely linear, simplistic view obscures exactly the kind of decision points that matter most for understanding why a trajectory unfolded the way it did, making a reviewer work considerably harder to reconstruct something the raw trace data already contained the whole time.

How to instrument tool calls specifically for observability

Each individual tool call an agent makes needs to be captured with enough structured detail to diagnose a problem specifically located at that exact step, the exact, precise parameters the agent passed, the exact raw result the tool returned, and whether that call succeeded, failed, or returned a result the agent’s subsequent reasoning treated as unexpected or problematic. This kind of structured, careful capture matters considerably more than it might initially, superficially seem to at first, since a tool call that technically succeeded, returning a response without any error, can still have returned a result that was substantively unhelpful or actively misleading for the agent’s actual task, a distinction only visible if the trace captures the tool’s actual returned content rather than merely its bare success or failure status alone.

Instrumenting tool calls well also means capturing enough context about why the agent chose to make that call in the first place, ideally alongside the model’s brief reasoning for that choice, since a tool call viewed purely in isolation, without the reasoning that led to it, tells a reviewer what the agent did but not why, and understanding why matters enormously for distinguishing a reasonable choice that happened to fail from a fundamentally poor choice that reflects a deeper problem in the agent’s decision-making.

How to capture and represent an agent’s reasoning and planning

Many modern agents produce some form of explicit intermediate reasoning, a rationale for why an action was chosen, before taking that action, and capturing this reasoning as its distinct element within a trace, rather than discarding it once the corresponding action has been taken, provides essential diagnostic value that the action sequence alone can’t fully provide. An agent’s final action can look entirely reasonable in isolation while its accompanying reasoning reveals a fundamentally flawed understanding of the task that happened, through some combination of luck or coincidence, to still produce an acceptable outcome on this particular occasion, exactly the kind of fragile, coincidental success a reviewer would never detect from the action sequence alone.

Representing this reasoning well in a trace means keeping it explicitly linked to the action it led to, rather than presenting reasoning and action as separate, disconnected streams a reviewer has to manually correlate themselves, and means preserving it in a form that’s readable, not buried inside an unstructured dump of the full underlying model prompt and response that a reviewer would have to manually parse to extract the relevant reasoning from considerable surrounding noise.

How observability helps distinguish a planning failure from an execution failure

When an agent trajectory ends in a failure, the actual root cause can sit at fundamentally different points along that trajectory, a flawed initial plan that was never going to succeed regardless of how well it was executed, or a sound plan that failed specifically because one particular step along the way, a tool call, a piece of retrieved information, didn’t behave as the agent had reasonably expected, and distinguishing between these two different failure categories matters enormously for knowing what to fix, improving the agent’s planning capability, versus improving the reliability of a tool or data source it depends on.

Good agent observability makes this distinction directly visible by preserving the agent’s original plan alongside the trajectory’s actual execution, so a reviewer can directly compare what the agent originally intended against what happened step by step, immediately seeing whether a failure traces back to the plan itself being fundamentally unsound, or to an execution step deviating from an otherwise perfectly reasonable plan. Without this explicit plan-versus-execution comparison, a reviewer is left inferring the agent’s original intent purely from its actions after the fact, a considerably less reliable and more error-prone way to diagnose which of these two different failure categories occurred.

How to trace across multiple agents in a multi-agent system

Systems where a primary agent delegates subtasks to specialized sub-agents, or where several agents collaborate on different parts of one larger overall task, introduce a distinct observability challenge beyond what a single-agent trace already needs to capture, since a failure in this kind of system can originate not just within any one individual agent’s trajectory but specifically in the handoff between two agents, one agent’s output being misinterpreted or incorrectly formatted for the next agent receiving it, a category of failure invisible to any observability approach that only ever traces each individual agent in isolation from the others.

Building effective multi-agent observability means capturing not just each individual agent’s internal trace but the handoffs between agents as their distinct, first-class elements, what exactly one agent passed to another, and how the receiving agent interpreted and used what it received, so a reviewer can directly inspect the handoff itself as a potential point of failure rather than only being able to inspect each agent’s internal behavior separately and having to manually infer whether the connection between them was the underlying problem.

How sampling works for agent traces at scale

Capturing a complete, detailed trace for every single agent trajectory a system handles becomes expensive at production volume, both in raw storage cost and in the sheer processing overhead of generating that level of detail continuously, which means most production agent observability practices rely on some form of deliberate sampling, capturing full, detailed traces for a representative subset of trajectories rather than every single one, the same fundamental trade-off that applies to evaluation and general monitoring sampling more broadly, applied here specifically to the considerably more data-intensive case of complete agent trace capture.

Effective trace sampling for agents specifically weights toward the categories of trajectory that carry the most diagnostic value, deliberately over-sampling trajectories that ended in a detected failure, trajectories that took an unusually large number of steps relative to similar tasks, and trajectories flagged by any other monitoring signal as potentially anomalous, rather than sampling purely at random, since a random sample systematically under-represents exactly the kind of rare, consequential trajectories a team most needs detailed trace data to understand and diagnose.

How agent observability supports root-cause debugging of a failure

When a trajectory is flagged as problematic, whether through an explicit user complaint, an evaluation review, or an automated anomaly signal, effective agent observability lets a team go from that flag directly to a complete, detailed reconstruction of exactly what happened during that trajectory, without needing to manually piece together fragments from several disconnected logging systems that each captured only part of the overall picture. This end-to-end traceability is precisely what turns agent observability from a passive record-keeping exercise into an active useful debugging tool a team can rely on when a problem needs to be understood quickly.

The most effective debugging workflows built on top of agent observability let a reviewer move fluidly between a high-level view of the trajectory’s overall shape and the full, granular detail of any step within it, starting from a broad understanding of roughly where things went wrong before drilling into the exact tool call, exact reasoning, or exact intermediate result that explains why. A team without this kind of fluid navigation between levels of detail typically ends up either stuck at too high a level to pinpoint a root cause, or drowning in undifferentiated raw detail without any efficient way to first narrow down where within a long trajectory to start looking.

How observability data feeds back into evaluation and benchmark design

Patterns that emerge repeatedly across many individual traces, a category of tool call that fails unusually often, a kind of task that reliably produces unusually long, inefficient trajectories, represent exactly the kind of concrete signal that should directly inform what an evaluation practice specifically tests for going forward, closing the loop between observability, which reveals what’s happening in trajectories, and evaluation, which measures whether a system’s behavior meets an acceptable bar, so that a pattern observability first surfaces gets explicitly built into the evaluation practice’s ongoing test coverage rather than being discovered anew each time it happens to recur.

This same feedback loop extends specifically to benchmark and test-case design, since a difficult or edge-case scenario that observability reveals occurring in production use is exactly the kind of scenario a benchmark or test suite should specifically include going forward, a considerably more grounded and realistic source of difficult test cases than scenarios a team merely imagines in advance without the benefit of observed trajectory data showing what challenging situations look like in practice.

How to build effective agent trace visualization tools and UIs

An effective trace visualization tool has to serve two meaningfully different use cases well, quick, high-level triage, letting a reviewer rapidly scan many trajectories to identify which ones warrant a closer look, and deep, detailed investigation, letting a reviewer who has already identified a problematic trajectory drill all the way down into its full, granular detail, and building a single tool that serves both use cases well requires deliberate, careful design rather than simply dumping raw trace data onto a screen and hoping it turns out to be usable in practice.

The most effective visualization tools support this dual purpose through a layered interface, a compact, scannable summary view for triage that surfaces just the key signals, did it succeed, how many steps did it take, was cost or latency unusual, alongside a considerably more detailed drill-down view reachable with a single action once a reviewer has identified a trajectory worth deeper investigation. A tool that only offers one of these two views, either forcing every review through exhaustive full detail or only ever offering a shallow summary with no path to deeper investigation, will frustrate whichever use case its missing view was needed for.

How agent observability handles sensitive data in traces

A complete agent trace, by its very nature, captures considerable detail about what happened during an interaction, which frequently means it also captures sensitive information, personal details a user shared, proprietary business data a tool call retrieved, information that needs careful handling rather than simply being stored and displayed with the same casual treatment a team might apply to routine, non-sensitive operational logs. Treating trace data with the same privacy and access-control discipline an organization already applies to its other systems handling sensitive information is essential, not optional, and a team that builds detailed trace capture without this discipline risks creating a serious privacy exposure precisely because trace data, by design, captures more raw, unfiltered detail than most other operational data a system generates.

Building this discipline in practice means applying redaction or masking to sensitive fields within a trace before that trace becomes broadly accessible for review, restricting full, unredacted trace access to the people who need it for legitimate debugging purposes, and applying the same kind of deliberate retention policy that matters for monitoring data more broadly, not keeping every detailed trace indefinitely at full, unredacted granularity longer than its diagnostic value justifies given the privacy exposure it represents.

How observability differs for long-running and asynchronous agents

An agent working through a task that spans hours or days, checking back periodically rather than completing everything in one continuous execution, introduces observability challenges a shorter-trajectory system doesn’t have to contend with, since a trace for this kind of extended, asynchronous trajectory has to remain coherent and navigable across a considerably longer span of time, and has to represent the agent’s state during the periods it wasn’t actively executing, not just the discrete moments when it took a visible action.

Building observability for this kind of long-running trajectory well means capturing not just the sequence of discrete actions but the evolving state the agent is maintaining across its extended task, what it currently believes has been accomplished, what it still considers outstanding, so a reviewer inspecting an in-progress or completed long-running trajectory can understand its full context rather than only seeing a series of disconnected action snapshots without the continuity that connects them together as one coherent, extended undertaking. A team whose observability tooling was built entirely around shorter, quickly-resolving trajectories often finds that tooling breaks down, or becomes considerably less useful, once applied to this kind of extended, asynchronous agent work.

How agent observability connects to production monitoring and cost monitoring as siblings

Agent observability, production monitoring, and cost monitoring each provide a distinct, complementary lens on a deployed agent’s behavior, and the three work best when connected rather than run as entirely separate, disconnected efforts. Production monitoring’s aggregate operational signals, an overall spike in latency or error rate, tell a team something is wrong somewhere, cost monitoring’s granular attribution tells a team where spend is concentrated, and agent observability’s detailed trace capture is what lets a team go from either of those aggregate signals down to a concrete trajectory that explains precisely why, closing the loop from a high-level anomaly signal to the granular root cause that explains it.

A team that builds agent observability entirely disconnected from its broader monitoring practice loses this connective value, ending up with detailed trace data that’s technically available but never gets consulted at the moment a monitoring alert fires, because the path from that alert to the relevant trace wasn’t ever built. The most effective practices wire these three together explicitly, a monitoring alert or a cost anomaly should link directly to the trajectories that plausibly explain it, letting a team move fluidly from noticing that something is wrong to understanding, at the level of a trajectory, why.

How agent observability should represent a trajectory’s use of retrieved context

An agent that draws on retrieved documents, whether through a dedicated retrieval step or through a tool call that queries some external knowledge source, introduces its observability requirement beyond the general tool-call instrumentation already covered, since the actual content of what was retrieved, not just the fact that a retrieval happened, matters enormously for understanding why the agent’s subsequent reasoning and actions turned out the way they did. A trace that records only that a retrieval call occurred, without preserving the passages returned, leaves a reviewer unable to distinguish between an agent that reasoned poorly from good retrieved material and one that reasoned entirely reasonably from retrieved material that was itself irrelevant or misleading.

This means agent observability specifically built around retrieval-augmented agents needs to preserve retrieved content as its explicit, inspectable element within the trace, linked directly to whatever downstream reasoning and action drew on it, mirroring the same kind of faithfulness-relevant detail that matters for dedicated RAG evaluation, but captured here for the purpose of trajectory-level debugging rather than aggregate quality measurement. A team building agent observability without this retrieval-content capture will find itself unable to answer one of the most common diagnostic questions a retrieval-augmented agent’s failure raises, whether the problem originated in what was retrieved or in what the agent subsequently did with it.

How to build alerting directly on top of observability signals rather than only aggregate metrics

Beyond the aggregate, system-wide alerting that belongs to broader production monitoring, agent observability data itself can support a more targeted category of alert, one built directly on patterns visible only at the level of individual trajectories, an agent that took an unusually high number of steps relative to similar tasks, a trajectory where the agent’s plan changed an unusually large number of times, a trajectory that involved a tool call to a sensitive capability that should trigger scrutiny regardless of whether the overall trajectory technically succeeded. These trajectory-level signals catch a distinct category of concern that aggregate metrics, averaged across an entire system’s traffic, structurally cannot surface on their own.

Building this kind of observability-driven alerting well means treating trajectory-level patterns as their first-class signals worth monitoring continuously, not just as something a reviewer might happen to notice while manually inspecting an individual trace during an unrelated investigation, and routing concerning patterns, an unusual step count, an unexpected sensitive tool call, directly to the same alerting and incident-response process that handles other production concerns, rather than letting this category of signal exist only as passive data a team would have to go out of its way to notice.

How agent observability should be organized and made searchable across many trajectories

A single, well-instrumented trace is useful for investigating one trajectory a reviewer already knows to look at, but diagnostic value at scale also depends on being able to search and filter across many trajectories at once, finding every trajectory that called a tool, every trajectory that took more than a certain number of steps, every trajectory associated with a user or feature, since the most useful diagnostic questions often start not with one known trajectory but with a pattern a team suspects might exist across many trajectories and needs to confirm or refute with aggregated trace data.

Building this searchability well means indexing trace data along the same structured dimensions that matter for the rest of an observability and monitoring practice, by feature, by tool used, by outcome, by step count, so that a team investigating a suspected pattern can query across the full population of recent trajectories rather than being limited to inspecting whatever individual traces happen to already be open in front of them. A team whose trace data exists only as individually retrievable records, without this cross-trajectory searchability, ends up unable to validate whether a problem they’ve noticed in one trajectory is an isolated incident or a systemic pattern worth dedicated attention.

How agent observability supports comparing behavior across different agent versions

When an agent’s underlying prompt, tool set, or underlying model changes, one of the most useful things observability can support is a direct, side-by-side comparison of how the new version behaves on the same or comparable tasks relative to the version it’s replacing, going beyond simply comparing aggregate success rates to comparing the shape of the trajectories themselves, whether the new version takes a different number of steps, calls different tools, or arrives at its outcomes through a different reasoning path even when the final result looks superficially similar.

This kind of version-to-version trajectory comparison catches a category of change that a pure before-and-after evaluation score, covered in the broader discussion of ongoing agent evaluation, can miss entirely, a new version that maintains an identical success rate while achieving it through a considerably less efficient or considerably more fragile-looking process, a degradation in the underlying quality of the agent’s actual reasoning that simply hasn’t yet shown up as a measurable drop in outcome-level success. A team building observability specifically to support this kind of version comparison gains an early, structural warning signal for exactly this kind of quality drift, well before it would eventually surface as a decline in the outcome-level metrics evaluation more broadly tracks.

Common mistakes teams make around agent observability

Several patterns recur often enough across teams building agent observability that naming them directly, explicitly is worth doing before they lead to a costly gap in understanding why a trajectory failed.

1. Relying on general-purpose logging built for conventional software, missing the branching, adaptive structure an agent trajectory requires to be usefully represented.

2. Capturing only an agent’s final output without the intermediate plan, reasoning, and tool calls, leaving a trace with essentially nothing to diagnose a failure against at all.

3. Presenting a trajectory’s full trace as an undifferentiated wall of raw text rather than a navigable, well-layered visualization a reviewer can work with.

4. Capturing tool call success or failure status without the actual returned content, missing technically successful calls that returned substantively unhelpful or misleading results.

5. Discarding an agent’s intermediate reasoning once its corresponding action has already been taken, losing the diagnostic value that reasoning provides.

6. Failing to preserve an agent’s original plan alongside its actual execution, making it considerably harder to distinguish a planning failure from an execution failure.

7. Tracing each individual agent in a multi-agent system entirely in isolation, missing failures that originate specifically in the handoff between two agents rather than within either one alone.

8. Sampling traces purely at random rather than deliberately, intentionally over-sampling failed, unusually long, or otherwise flagged trajectories that carry the most diagnostic value.

9. Building trace visualization that only supports either high-level triage or deep, granular detail, frustrating whichever use case its missing view was needed for.

10. Treating trace data with considerably less privacy and access-control discipline than other systems handling sensitive information, despite traces capturing considerable raw, unfiltered detail.

11. Retaining every single detailed trace indefinitely at full granularity, accumulating privacy exposure well beyond what diagnostic value justifies.

12. Building observability tooling entirely around short, quickly-resolving trajectories, leaving long-running, asynchronous agent work with inadequate, considerably incoherent trace coverage.

13. Never feeding patterns observability surfaces back into evaluation and benchmark design, rediscovering the exact same recurring failure mode repeatedly rather than building it into ongoing test coverage.

14. Running agent observability entirely, completely disconnected from broader production and cost monitoring, losing the direct path from an aggregate anomaly signal to the trajectory that explains it.

15. Treating agent observability as a passive record-keeping exercise rather than an active debugging tool a team navigates fluidly between high-level and granular detail.

16. Capturing merely that a retrieval call occurred without preserving the actual retrieved content, leaving a reviewer unable to tell whether a failure originated in retrieval or in subsequent reasoning.

17. Building alerting only on aggregate, system-wide metrics, missing trajectory-level patterns like unusual step counts or unexpected sensitive tool calls that aggregate metrics structurally cannot surface.

18. Storing trace data only as individually retrievable records, without the cross-trajectory searchability needed to confirm whether a suspected pattern is isolated or systemic.

19. Comparing agent versions purely on aggregate success rate, missing a new version that maintains identical outcomes while reasoning through a considerably less efficient or more fragile underlying process.

What connects all nineteen of these mistakes is a single underlying pattern: treating an agent’s trajectory as though it were simply a longer, more elaborate version of a conventional software request, when it’s a branching, adaptive sequence of plans, actions, retrieved context, and reasoning that requires its dedicated data model and its dedicated, purpose-built visualization to be understood, rather than something general-purpose logging and monitoring tooling, built for a considerably more linear kind of system, was ever designed to represent well in the first place.

The deeper principle underneath all of this is that an agent’s behavior only becomes truly understandable once its full trajectory, plan, actions, reasoning, retrieved context, and outcome together, is captured and made navigable as one coherent, connected whole, searchable across many trajectories and comparable across versions, and a team that only ever sees fragments of that trajectory, an aggregate metric here, a final output there, is fundamentally limited in how well it can explain why an interaction unfolded the way it did, no matter how sophisticated its other monitoring and evaluation practices happen to otherwise be.