What is agent tracing?

Summary slides

Why agent tracing is a distinct technical practice from general LLM tracing and agent observability

General LLM tracing, covered in its dedicated discussion, provides the foundational span-and-trace infrastructure that captures any individual model call, its prompt, its completion, its timing, entirely regardless of what kind of application that particular call happens to be part of. Agent observability, covered separately in its own discussion, focuses on how a team uses and analyzes the resulting trace data, visualizing a trajectory, diagnosing a failure, comparing different versions. Agent tracing sits specifically between these two, the actual technical work of instrumenting an agent’s particular execution loop so that the general tracing infrastructure captures the structure an agent produces, its recurring think-act-observe cycle, its evolving internal state, its tool-selection reasoning, rather than only ever capturing a flat, undifferentiated sequence of individual model calls that merely happen to have occurred in some particular, arbitrary order.

This distinction matters because an agent’s execution loop has structural properties that a naive application of generic tracing infrastructure simply won’t automatically capture correctly, the same underlying tool might get called multiple times across a trajectory with different arguments each time, an agent’s internal state accumulates and mutates across steps in ways that a single isolated span doesn’t naturally represent, and the actual decision of which tool to call next is itself a distinct traceable event entirely separate from the tool call it eventually leads to. Agent tracing is the discipline of building instrumentation that correctly captures all of this particular structure, providing the accurately, correctly shaped raw material that agent observability’s visualization and analysis tools depend on to work well in the end.

How the think-act-observe loop gets encoded as a trace structure

Most agent architectures follow some variation of a repeating, ongoing cycle, the agent reasons about what to do next, takes an action, typically a tool call, and then observes the result of that action before beginning the whole cycle again with that new information incorporated, and agent tracing represents this cycle explicitly by creating a distinct span for each of these three phases within every single iteration, a reasoning span capturing the agent’s internal deliberation, an action span capturing the tool call made, and an observation span capturing the result that action returned, all nested clearly underneath a parent span representing that iteration of the overall loop.

This explicit three-phase structure, rather than collapsing an entire iteration into one single, undifferentiated span, is what lets a reviewer or an automated analysis tool distinguish between different categories of problem within the exact same iteration, a reasoning failure where the agent’s deliberation was flawed even before any action was taken, an action failure where the reasoning was sound but its execution went wrong, or an observation-handling failure where the agent received a correct result but then misinterpreted or mishandled it in the subsequent iteration. Collapsing these three phases into one span loses exactly this diagnostic granularity, forcing a reviewer to manually untangle which phase within an iteration went wrong.

How agent tracing captures state mutations between steps

An agent typically maintains some form of internal state across its trajectory, a working memory of what it has learned, a scratchpad of intermediate results, a running summary of progress so far, and this state doesn’t stay fixed, it mutates at each step as the agent incorporates new information, and agent tracing needs to capture these mutations explicitly, recording the state both before and after each individual step, rather than only capturing the agent’s final output state at the very end of a completed trajectory. A trace that only shows the agent’s state at the beginning and the end of a trajectory, without the intermediate mutations along the way, gives a reviewer no way to pinpoint the step where a piece of state was incorrectly updated, incorrectly dropped, or incorrectly overwritten.

Capturing this state evolution well means treating state mutation as its distinct, explicit trace element, linked directly to the step that caused it, so a reviewer investigating a trajectory that ended with an agent operating on incorrect or stale information can trace that corruption directly back to the exact step where it first occurred, rather than only knowing that the final state was wrong without any principled way to identify precisely when and how it became wrong in the first place.

How to trace agents built on different frameworks with different execution models

Different agent frameworks structure their execution loops in different ways, some built around an explicit, single-threaded reasoning loop, others built around a more event-driven or graph-based execution model where an agent’s next step gets determined by which node or edge in a defined execution graph fires next, and agent tracing has to adapt its instrumentation approach to whatever particular execution model the underlying framework uses, rather than assuming every agent follows the exact same simple, linear think-act-observe structure covered in the more general discussion above.

This means effective agent tracing infrastructure typically provides framework-instrumentation hooks, tapping into whatever callback or event system a framework already exposes for its internal execution model, translating that framework’s particular internal events into the standardized span structure the broader tracing infrastructure expects, rather than requiring every team to build entirely bespoke, one-off instrumentation from scratch for each framework it happens to use. A team building or adopting agent tracing tooling should specifically check whether that tooling understands the particular execution model its chosen agent framework uses, since a tracing approach built assuming one execution model can produce misleading or incomplete traces when applied naively to a framework built around a meaningfully different one.

How tracing handles recursive or self-invoking agent calls

Some agent architectures allow an agent to invoke a fresh instance of itself, or a similar sub-agent, to handle a subtask, a pattern that introduces a distinct tracing challenge beyond the simpler, single-agent case, since a naive tracing implementation can struggle to correctly distinguish between the outer, original agent’s trace and the inner, recursively invoked agent’s separate trace, especially when both are, structurally, running the exact same underlying code and would otherwise be difficult to tell apart from trace data alone without some explicit, deliberate marker distinguishing them.

Handling this well means explicitly tagging each recursive invocation with its distinct identifier and its explicit depth level within the overall recursive structure, so a reviewer inspecting the resulting trace can clearly see not just that a recursive call occurred but specifically how deep that recursion went and which particular invocation, at which particular depth, is responsible for which particular part of the overall combined trace. A tracing implementation that doesn’t explicitly handle this recursive case tends to produce traces that become difficult, or in some cases functionally impossible, to correctly interpret once recursion beyond a shallow, single level starts occurring in practice.

How to trace asynchronous and concurrent tool execution within a single agent step

An agent step that involves calling several independent tools concurrently, rather than one tool at a time in strict sequence, introduces its distinct tracing requirement, since the resulting spans for those concurrent calls need to be represented as parallel children of the same parent step, with their independent timing, rather than being forced into an artificially sequential representation that would misleadingly suggest one call happened before another when they occurred simultaneously, in parallel, within the exact same step.

This matters for accurate diagnosis specifically because a step’s overall latency, when several tool calls happened concurrently, is determined by whichever single call took the longest, not by the sum of all the calls’ individual durations, and a trace that incorrectly represents concurrent calls as sequential would produce a misleadingly inflated impression of that step’s actual latency, one that doesn’t correspond to what a user experienced. Correctly representing this concurrency in the trace structure itself, rather than only in some separate, informal documentation about how the code happens to work, is what lets the latency-breakdown analysis covered in the broader discussion of LLM tracing produce accurate, trustworthy results for agent steps specifically involving this kind of concurrent execution.

How agent tracing integrates with a framework’s callback and hook system

Most mature agent frameworks already expose some form of callback or hook system, letting external code register to be notified when a step begins, when a tool gets called, when the agent’s reasoning produces an intermediate result, and building effective agent tracing typically means integrating directly with these existing hooks rather than requiring a team to modify an agent’s core execution logic just to add tracing instrumentation on top of it. This integration approach keeps tracing decoupled from an agent’s actual business logic, meaning a team can add, remove, or modify its tracing instrumentation without touching the underlying agent code that instrumentation is observing.

The trade-off worth understanding here is that a framework’s exposed hooks may not always provide quite the level of granularity or quite the timing a team’s tracing needs, a hook that fires only after an entire step has fully completed, for instance, can’t directly support the kind of fine-grained, phase-level tracing covered earlier in this discussion, which sometimes means effective agent tracing requires either extending a framework’s hook system with additional, custom hooks, or, in cases where a framework’s hook system is simply too limited, falling back to some of the more direct, manual instrumentation approaches covered in the broader discussion of LLM tracing.

How to use traces to replay or simulate a past agent run

A complete, well-structured agent trace contains everything technically needed to attempt reconstructing what happened during a past run, the inputs at each step, the tool results received, the reasoning that led to each action, and this reconstruction capability, called replay, provides valuable debugging power beyond simply reading through a static trace, letting a developer step through a past trajectory interactively, inspecting intermediate state at whatever point in the sequence they want to examine more closely, rather than only reading a fixed, static record from start to finish.

Building useful replay capability on top of trace data means the underlying trace itself has to capture enough complete, faithful detail to support this kind of reconstruction, every tool result, every piece of state, not just a lossy summary of what happened, since a replay tool can only ever be as complete and as faithful as the underlying trace data it’s working from. A team that captures only partial, summarized trace detail will find its replay tooling correspondingly limited, able to show only a similarly partial, incomplete reconstruction of what a past run did.

Why exact replay is fundamentally limited by non-determinism

Re-running an agent’s exact original sequence of model calls, expecting to reproduce the exact same original results, runs directly into the same non-determinism covered in the broader discussion of AI system testing, since a model call made today, even using the exact identical prompt that was originally used, can produce a different result than it did when that original trajectory first ran, which means agent trace replay, in the fullest, most rigorous sense, can never guarantee it’s reproducing the exact same original trajectory rather than merely a plausible, structurally similar one.

This limitation shapes what agent trace replay can realistically be used for in practice: useful for reviewing exactly what happened in a past run, using the actual historical data captured at the time; useful for testing how a modified version of an agent would handle the exact same original inputs and previously observed tool results; but not reliable as a way to deterministically reproduce an exact original trajectory by re-executing live model calls from scratch. A team building replay tooling should be explicit and clear about which of these two meaningfully different use cases its tooling is built to support, since conflating them leads to misplaced confidence in exactly what a replay demonstrates.

How to trace an agent’s internal tool-selection decision

Before an agent calls a tool, it typically has to decide which tool, among potentially several available options, is the right one to use for the current step, and this decision, the tool-selection reasoning itself, is a distinct, separately traceable event worth capturing explicitly, separate from the resulting tool call it eventually leads to, since a reviewer investigating a failure caused by the agent choosing an inappropriate tool needs to see the reasoning behind that choice, not just the fact that a particular tool call was ultimately, eventually made.

Capturing this tool-selection reasoning as its explicit trace element also supports a valuable analytical use case beyond individual-trajectory debugging, letting a team aggregate tool-selection decisions across many separate trajectories to understand which tools an agent tends to choose correctly versus incorrectly in aggregate, informing decisions about which tool descriptions might need clarification, or which tools might be redundant or confusingly, unhelpfully similar to each other from the agent’s actual decision-making perspective.

How agent tracing captures the evolving state of memory and scratchpad across a run

Beyond the step-level state mutations covered earlier, many agents maintain a longer-lived working memory or scratchpad that persists and accumulates across an entire trajectory, sometimes even across multiple separate but related trajectories in systems built around some form of longer-term memory, and tracing this evolving memory well means capturing it as its distinct, continuously evolving trace element that a reviewer can inspect at any point in a trajectory’s timeline, rather than only ever seeing the memory’s final accumulated state once a trajectory has completely finished.

This continuous memory tracking connects directly to the long-running, asynchronous agent observability covered in the broader discussion of agent observability more generally, since a long-running agent’s memory state is precisely the kind of evolving context a reviewer needs visibility into to understand that agent’s current behavior at any point during its considerable, extended execution, not merely at its eventual final completion once the entire trajectory has fully run its course.

How to version and diff agent traces across code changes

When an agent’s underlying code, its prompts, its available tools, its reasoning logic, changes between two versions, comparing traces produced by the old version against traces produced by the new version, for the exact same or equivalent inputs, provides direct, concrete evidence of exactly how that code change affected the agent’s behavior, extending the version comparison capability covered in the broader discussion of agent observability with the technical machinery needed to diff two traces meaningfully at the level of their individual, corresponding steps.

Building this diffing capability well means aligning corresponding steps across the two traces being compared, even when the exact number or exact order of steps has changed between versions, and highlighting specifically where the two traces diverge, a tool call that appears in one version’s trace but not the other, a reasoning step that reaches a different conclusion given what should be equivalent input, rather than only being able to compare the two traces’ final outputs in isolation without any visibility into the granular path each version took to reach that final outcome.

How tracing supports testing agents against recorded historical traces

A library of historical traces, particularly ones capturing especially challenging or edge-case trajectories, provides a valuable resource for testing a new agent version, replaying the historical inputs and previously observed tool results from a captured past trace and checking whether the new version handles that same challenging scenario at least as well as, or ideally meaningfully better than, the original version did when that trace was first captured, connecting directly to the discussion of how observability data feeds back into evaluation and benchmark design covered in the broader agent observability discussion.

This trace-based testing approach provides a more realistic, grounded source of test scenarios than test cases a team merely imagines in advance, since a historical trace, by definition, represents something that happened in usage rather than a scenario someone merely speculated might theoretically occur, and building this kind of trace-based regression testing directly into a team’s ongoing development practice helps ensure that a difficult scenario, once encountered and correctly handled, doesn’t quietly regress and start failing again in some later version of the agent without anyone specifically noticing.

How agent tracing data should be structured for downstream automated analysis

Beyond supporting direct human review, well-structured agent trace data should also support automated, programmatic analysis, computing aggregate statistics across many trajectories, feeding trajectory data into the kind of automated evaluation covered in the broader discussion of agent evaluation, and detecting the trajectory-level anomaly patterns covered in the discussion of agent observability’s alerting, all of which depend on trace data being structured consistently and predictably enough for automated tooling to parse and analyze it reliably, rather than being structured purely around what happens to be convenient or readable for a human reviewer looking at one trace at a time.

This means agent tracing infrastructure benefits from adopting a consistent, well-documented schema for its trace structure field names and data types for reasoning spans, action spans, and state mutations, applied consistently across every single trajectory a system produces, rather than letting that structure drift or vary inconsistently across different parts of an application or across different points in a system’s ongoing development over time. A team whose trace structure has drifted inconsistently across its codebase finds its downstream automated analysis tooling correspondingly brittle, requiring constant, ongoing adjustment to handle each new inconsistency as it happens to be discovered.

How agent tracing should handle human-in-the-loop interruptions within a trajectory

Many agent deployments don’t run entirely autonomously from start to finish, they pause at a point to request explicit human approval before taking a particularly consequential action, or to ask a clarifying question a human then answers before the trajectory continues, and this human-in-the-loop pattern introduces a distinct tracing requirement beyond the purely machine-driven steps covered so far, since the resulting trace needs to explicitly represent not just what the agent itself did but the point where execution paused, what was asked of the human, how long that pause lasted, and what the human’s response was, treating this human interaction as its distinct, first-class span rather than an invisible gap in an otherwise purely automated trace.

Capturing this well matters because the latency a human-in-the-loop pause contributes to a trajectory’s overall total duration is a different kind of latency than the model or tool latency covered elsewhere in this discussion, driven by human response time rather than any technical processing delay, and conflating the two produces a misleading latency breakdown that would incorrectly suggest a technical performance problem when the actual cause was simply a human taking a reasonable amount of time to consider and respond to a legitimate request for their input. A trace that explicitly separates human-response latency from model and tool latency lets a team correctly interpret an unusually long trajectory duration rather than mistakenly investigating it as though it were a technical performance issue.

How to trace an agent’s error-handling and recovery logic explicitly

When a step within an agent’s trajectory fails, a tool call errors out, a model call times out, well-built agents typically don’t simply halt entirely, they execute some form of deliberate recovery logic, retrying the failed step, falling back to an alternative approach, or explicitly acknowledging the failure and adjusting the remaining plan accordingly, and agent tracing needs to represent this recovery logic as its explicit, distinct element within the trace, clearly linked to the failure that triggered it, rather than letting the recovery attempt simply blend into the trace as though it were an entirely ordinary, unremarkable step indistinguishable from the rest of the trajectory.

Representing recovery logic explicitly this way lets a reviewer directly see not just that a trajectory eventually succeeded despite an underlying failure occurring somewhere along the way, but specifically how that success was achieved, through a successful retry, through a fallback to an alternative approach, through a graceful, deliberate adjustment to the remaining plan, an important distinction for judging whether an agent’s recovery behavior is working well or whether it’s succeeding only through a kind of fragile, coincidental luck that a differently-shaped failure might not survive the next time a similar situation arises. A trace that hides recovery logic inside what looks like an ordinary, unremarkable step leaves this important distinction invisible to anyone reviewing that trace after the fact.

How to manage the storage overhead detailed agent tracing introduces

The rich, structural detail agent tracing captures, per-phase spans, state mutations, tool-selection reasoning, memory snapshots, recovery logic, human-in-the-loop interactions, considerably increases the actual storage volume a trace generates compared to a simpler, flatter trace that only records individual calls without this additional structural detail, and a team building comprehensive agent tracing needs to specifically weigh this storage cost against the diagnostic value that additional structural detail provides, applying the same kind of deliberate capture decisions and sampling strategy covered in the broader discussions of LLM tracing and agent observability, but specifically calibrated to account for agent tracing’s considerably richer, more structurally detailed trace format.

This weighing decision often means applying differentiated retention specifically by trace element type rather than a single, uniform retention policy applied identically across an entire trace, retaining reasoning and tool-selection detail, which tends to carry the most diagnostic value for understanding why a trajectory unfolded the way it did, for a somewhat longer period than raw, verbose tool-call output, which can often be usefully summarized or compressed after some shorter initial retention window without losing much of its diagnostic value. A team that applies one single, undifferentiated retention policy uniformly across every element of a rich agent trace either retains far more raw detail than it can afford to store at scale, or discards valuable structural detail prematurely in an attempt to control that same storage cost.

How agent tracing should represent an agent’s confidence or uncertainty at each step

Some agent architectures produce an explicit signal of how confident the model itself is in a reasoning step or a tool-selection decision, and when this signal is available, capturing it as its explicit element within the trace provides valuable additional diagnostic context beyond the bare action or decision itself, letting a reviewer distinguish between a trajectory that went wrong despite the agent expressing high confidence at every step, suggesting a deeper reasoning problem worth investigation, and one where the agent’s expressed uncertainty at the point where things went wrong at least correctly signaled that a problem was more likely at that particular step.

This confidence signal connects directly to the broader discussion of honesty and calibration covered in the discussion of LLM evaluation, extending that same evaluative concern down to the level of an individual trace, providing the granular evidence needed to assess whether an agent’s expressed confidence, at the level of individual steps within a trajectory tracks its reliability at each of those steps, rather than only being able to assess calibration in the aggregate across many separate, disconnected interactions without this kind of step-level, within-trajectory granularity.

Common mistakes teams make around agent tracing

Several patterns recur often enough across teams instrumenting agents for tracing that naming them directly, explicitly is worth doing before they lead to a costly gap in the raw trace data that both agent observability and downstream automated analysis depend on to function well.

1. Collapsing an entire think-act-observe iteration into one undifferentiated span, losing the ability to distinguish reasoning, action, and observation failures within the exact same step.

2. Capturing only an agent’s final state at the end of a trajectory, missing the intermediate mutations needed to pinpoint exactly when a piece of state first became corrupted.

3. Applying instrumentation built for one execution model uniformly to a framework built around a different one, producing misleading or considerably incomplete traces.

4. Failing to explicitly tag recursive or self-invoking agent calls with their identifier and actual depth, producing traces that become difficult to correctly interpret once recursion occurs.

5. Representing concurrent tool calls as artificially sequential within a trace, producing a misleadingly inflated impression of a step’s actual latency.

6. Modifying an agent’s core execution logic just to add tracing rather than integrating with the framework’s existing callback and hook system directly.

7. Building replay tooling entirely on top of partial, summarized trace data, limiting reconstruction to a similarly partial, incomplete picture of what a past run did.

8. Treating agent trace replay as a way to deterministically reproduce an exact original trajectory, rather than acknowledging the limits non-determinism imposes on that goal.

9. Capturing only the resulting tool call without the agent’s tool-selection reasoning that led to it, losing insight into why an inappropriate tool was chosen.

10. Capturing an evolving memory or scratchpad only at a trajectory’s final completion, losing visibility into that memory’s actual state at any earlier point during execution.

11. Comparing two agent versions’ traces only by their final outputs, without any way to diff the granular path each version took to reach that outcome.

12. Never building a library of historical traces into an ongoing regression testing practice, letting a previously solved, difficult scenario quietly regress without anyone noticing.

13. Letting trace structure drift inconsistently across different parts of a codebase, producing downstream automated analysis tooling that’s correspondingly brittle and unreliable.

14. Treating agent tracing as effectively identical to general LLM tracing, missing the structural elements, state mutation, tool selection recursion, an agent’s execution requires.

15. Building agent tracing infrastructure without ever validating it against a complex trajectory involving recursion, concurrency, or long-running memory to confirm it holds up.

16. Treating a human-in-the-loop pause as an invisible gap rather than an explicit span, conflating human response latency with model and tool processing latency.

17. Letting an agent’s error-handling and recovery logic blend invisibly into an ordinary step, hiding whether a trajectory succeeded through resilience or fragile, coincidental luck.

18. Applying one single, undifferentiated retention policy uniformly across a rich agent trace, either over-retaining raw detail or discarding valuable structural information far too prematurely.

19. Discarding an agent’s available confidence signal at each individual step, losing the granular evidence needed to assess whether expressed confidence tracks reliability within a trajectory.

What connects all nineteen of these mistakes is a single underlying pattern: treating agent tracing as though it were simply general LLM tracing applied to a slightly more elaborate use case, when an agent’s distinct execution structure, its recurring loop, its evolving state, its tool-selection decisions, its potential recursion and concurrency requires its dedicated instrumentation approach specifically built to capture that particular structure accurately, rather than one that merely captures a flat, undifferentiated sequence of individual calls loosely connected in time.

The deeper principle underneath all of this is that an agent’s behavior is fundamentally a process, not merely a sequence of isolated events, and capturing that process, its evolving state, its internal decisions, its expressed confidence, its structural concurrency, recursion, human interruptions, and recovery logic, requires trace instrumentation deliberately built around the shape that process takes, rather than infrastructure originally designed for the considerably simpler case of one isolated, independent model call at a time, which is exactly, precisely why agent tracing deserves its dedicated technical attention distinct from both the general LLM tracing foundation beneath it and the higher-level observability practices built on top of it.