What is LLM tracing?

Summary slides

Why LLM tracing exists as a foundational practice underneath higher-level observability

Every other practice covered elsewhere in this broader discussion of observability, reconstructing an agent’s full trajectory, attributing cost to a feature, diagnosing a latency spike, ultimately, fundamentally depends on the exact same underlying raw material: a structured, reliable record of what happened at the level of individual model calls. Without that foundational record, agent observability has nothing to visualize, cost monitoring has nothing to attribute spend against, and evaluation has no ground truth record of what a system did during an interaction being reviewed. LLM tracing is precisely the practice that produces this foundational record, which is why it functions less as one observability technique among many and more as the infrastructure layer the other techniques are built on top of.

This foundational role is exactly why LLM tracing deserves its dedicated, careful treatment rather than simply being assumed to emerge automatically as an incidental side effect of building the higher-level practices that depend on it. A team that builds agent observability or cost attribution without first establishing solid, consistent tracing underneath finds those higher-level practices resting on an unreliable, shaky foundation, prone to gaps, inconsistencies, and missing detail that no amount of sophistication at the higher level can ever truly compensate for.

What a trace and a span mean in the LLM context

Borrowing directly, deliberately from the vocabulary of conventional distributed tracing, a span represents one discrete unit of actual work, in the LLM context, typically one individual call to a model, one tool invocation, or one retrieval operation, carefully capturing that unit’s start time, end time, and whatever input and output data is relevant to understanding what happened during it. A trace is the complete collection of spans that together represent one full, end-to-end interaction, from the moment a user’s request first entered the system to the moment a final response was returned, organized into a coherent structure that clearly shows how the individual, separate spans relate to and depend on each other throughout.

This span-and-trace vocabulary matters because it provides a shared, precise language for talking about LLM application behavior that maps directly onto tooling and conventions the broader software engineering field already has extensive, considerably mature experience working with, rather than requiring LLM applications to invent an entirely new, bespoke vocabulary of their own for describing what is, at its structural core, a similar underlying problem, understanding how a complex request moved through a system’s various component parts, that distributed tracing has already spent considerable, real time refining effective solutions for within the conventional software context.

How nested and chained LLM calls get represented as parent-child spans

A realistic, real-world LLM application rarely ever makes just one single, isolated model call, it typically involves a chain, an initial call that produces an intermediate result, which then gets fed into a subsequent call, sometimes with tool calls or retrieval steps interspersed directly between the individual model calls themselves, and representing this nested structure accurately means organizing spans into a proper parent-child hierarchy, where an overall request forms the root span, and each individual model call, tool call, or retrieval step within that request becomes a child span nested directly underneath it, or in the case of a call resulting directly from a previous call’s output, nested underneath that preceding span rather than merely flatly underneath the root span itself.

This hierarchical structure is precisely what makes a complex trace navigable rather than just a flat, undifferentiated list of individual calls arranged purely in chronological order, since a reviewer inspecting a trace needs to understand not just that ten different, separate spans occurred but specifically, precisely how they relate to each other overall, which calls were sequential steps within one coherent, connected chain, and which occurred entirely in parallel as independent, unrelated operations. A tracing implementation that flattens this hierarchy loses exactly the structural information that makes a complex, multi-call interaction comprehensible rather than merely enumerable at a surface level.

How tracing captures prompts, completions, and token counts at the span level

Beyond simple, basic timing information alone, a useful LLM span captures the substantive content of what happened during that call, the exact, precise prompt sent to the model, the exact completion the model returned, and the token counts for both categories, broken out separately, distinctly as input and output tokens given how differently those two categories tend to be priced by providers. This substantive capture, not merely recording that a call happened but recording what happened during it, is precisely what elevates tracing from a purely operational, performance-focused practice into something that also directly supports content-level debugging cost attribution, and evaluation.

Capturing this precise level of detail at the individual span level, rather than only ever at the level of a complete trace matters because it lets a reviewer or an automated analysis tool isolate exactly which call within a longer chain really consumed a disproportionate number of tokens, took unusually long, or produced a substantively unexpected result, precisely, exactly the granularity that a trace aggregated only at the whole-request level would lose entirely. This span-level granularity is exactly, precisely what makes tracing data useful as the raw foundation for the more cost attribution and trajectory reconstruction covered in the discussions of cost monitoring and agent observability more broadly across this collection.

How tracing differs from and complements distributed tracing in conventional software

Conventional distributed tracing tracking a request as it moves across multiple services in a larger system, shares structural similarity with LLM tracing, both are fundamentally about representing a complex operation as a structured hierarchy of timed spans, but LLM tracing extends this established concept with content specific to what makes an LLM call meaningfully different from a typical service call, the substantive prompt and completion content, token-based cost accounting, and the non-deterministic nature of what a call returns, none of which conventional distributed tracing tooling was originally built with in mind.

The most effective LLM tracing implementations don’t discard conventional distributed tracing’s established value, they extend it, representing LLM calls as a specialized kind of span within the exact same overall trace that also captures the surrounding conventional application logic, database queries, external API calls, that an LLM-powered application typically also involves. This unified approach means a team debugging a production issue can see the complete picture, both the LLM-detail and the surrounding conventional application behavior, within one single, coherent trace, rather than having to separately reconstruct and manually correlate two entirely disconnected observability systems.

How trace context propagation works across a multi-step request

For a trace to represent one coherent, connected interaction rather than a collection of disconnected, individually timed spans, the system needs a mechanism for propagating trace context, a shared identifier linking every span back to the same overall trace, across every component involved in handling a request, including across service boundaries in a system built from several separate, independently deployed components. Without reliable context propagation, a request that touches several different services ends up producing several separate, disconnected traces rather than one unified trace a reviewer can follow end to end.

This propagation typically works by passing a trace identifier explicitly through whatever mechanism the request itself is using to move between components, an HTTP header, a message queue’s metadata, ensuring that every component handling a piece of that request can correctly attach its spans to the same overall trace regardless of which service or process executed that particular step. Getting this propagation right across every boundary in a system is a non-trivial engineering effort, and a team that gets it wrong in even one place ends up with traces that mysteriously and silently fragment at exactly that boundary, a gap that’s often only discovered well after the fact, precisely when someone needs a complete trace to debug a problem.

How instrumentation gets added to an application

Adding tracing to an application generally happens through one of several distinct approaches, manual instrumentation, where a developer explicitly writes code to start and end a span around each operation worth tracing, SDK-level instrumentation, where a tracing library automatically wraps calls made through an LLM provider’s client library without requiring the developer to add explicit tracing code around every individual call, and broader auto-instrumentation, where a tracing agent automatically detects and instruments calls across an entire application with minimal developer effort required at all.

Each of these approaches carries a trade-off worth understanding clearly, manual instrumentation gives a developer the most precise, deliberate control over exactly what gets captured and how, at the cost of requiring deliberate, ongoing effort to add and maintain that instrumentation as an application evolves, while automatic instrumentation requires considerably less ongoing developer effort but can capture either too much irrelevant detail or too little of the meaningful detail a team needs for its particular debugging and analysis purposes. Most mature tracing setups end up using some blend of both, automatic instrumentation providing broad, baseline coverage across an entire application, supplemented by manual instrumentation added deliberately at the points where a team has identified a particular need for additional custom detail.

How standards like OpenTelemetry’s semantic conventions shape LLM tracing

OpenTelemetry, the widely adopted open standard for distributed tracing and observability more broadly, has extended its semantic conventions specifically to cover LLM and generative AI calls, standardizing exactly what attributes a well-formed LLM span should include, model name, token counts, the operation type, so that traces produced by different tools and different applications can be compared, aggregated, and analyzed using shared, common tooling rather than each team inventing its bespoke, incompatible trace format that only works with its custom-built analysis tools.

Adopting these shared standards matters considerably for interoperability, since a team using a standards-compliant tracing implementation can switch between different backend tools for storing and analyzing trace data without needing to rebuild its instrumentation from scratch each time, and can integrate LLM-trace data into the exact same broader observability platform that already handles the rest of its application’s conventional distributed tracing, rather than needing to maintain two entirely separate, disconnected observability systems that were never designed to work together in the first place.

How tracing handles streaming responses specifically

Many LLM applications stream a model’s response back to a user incrementally as it’s generated, rather than waiting for the complete response before returning anything at all, and this streaming behavior introduces a tracing challenge beyond what a simple, single-request, single-response call would require, since a span representing a streamed call needs to capture not just a single start and end time but the timing of the first token arriving, a metric called time-to-first-token that matters enormously for a user’s actual perceived responsiveness, distinct from the total time the complete response ultimately took to finish generating in full.

Tracing a streamed call well means capturing this time-to-first-token metric as its explicit, distinct data point within the span, alongside the total completion time, since these two metrics can diverge considerably and each one reveals something different about a system’s actual real-world performance, a model that’s slow to start generating but then streams quickly once it does start behaves very differently, from an actual user’s experience, than one that starts quickly but then streams considerably more slowly overall, even if both ultimately produce a complete response in roughly the same total elapsed time.

How tracing supports latency breakdown within a single LLM call

A single LLM call’s total observed latency is itself often composed of several distinct underlying phases, time spent in a request queue waiting to be processed, time the underlying model provider itself spent generating the actual response, and time spent on the application’s side processing that response before it becomes available to whatever calling code requested it, and useful tracing captures these distinct phases as their separate, explicit timing data within a span rather than collapsing everything into one single, undifferentiated latency number that obscures exactly where within that overall span the real time was spent.

This latency breakdown directly supports the more general latency-cause diagnosis covered in the broader discussion of production monitoring, providing the granular trace-level data that a monitoring dashboard’s aggregate latency-by-cause view is ultimately built from, extending that same diagnostic principle down to the level of one individual call rather than only an aggregate view spanning an entire system’s overall traffic. A team debugging one slow interaction needs exactly this granular, per-call latency breakdown, while a team tracking overall system health needs the aggregate view built by rolling up many individual spans’ worth of this same underlying detail.

How trace data gets exported, stored, and queried at scale

Trace data generated by an application’s instrumentation typically gets exported to a dedicated backend system specifically built for storing and querying this kind of structured, timestamped, hierarchical data at production volume, since the raw volume of spans a busy application generates quickly exceeds what a conventional application database was ever designed to handle efficiently. This backend system needs to support both the kind of targeted queries a reviewer investigating one particular known trace needs, and the kind of broader, aggregate queries needed for the cross-trajectory pattern analysis covered in the discussion of agent observability, finding every trace matching a set of conditions across a large representative population of recent traffic.

Choosing and configuring this backend well means understanding the trade-offs between different storage and query approaches, a backend optimized purely for fast, targeted single-trace lookup may perform considerably worse on broad, aggregate analytical queries, and vice versa, and a team’s actual usage pattern, how often it’s doing targeted single-trace debugging versus broad, aggregate pattern analysis, should inform which trade-offs matter most for its particular needs, rather than defaulting to whatever backend happens to be easiest to initially set up without considering how well it will serve the team’s ongoing query patterns at scale.

How tracing integrates with existing APM tooling

Most organizations building LLM-powered applications already have an existing application performance monitoring practice covering their conventional, non-LLM infrastructure, and the most effective LLM tracing implementations integrate directly into that existing APM tooling rather than standing entirely apart from it as a completely separate, parallel system a team has to learn and operate independently. This integration means LLM spans appear within the exact same traces, dashboards, and alerting systems that already cover the rest of an application’s conventional behavior, letting a team see the complete, unified picture of how a request moved through both its conventional application logic and its LLM-processing within one single, coherent view.

This integration matters because a realistic production incident rarely respects the boundary between LLM-and conventional application concerns cleanly, a slow response might trace back to conventional database latency, LLM provider latency, or some combination of both interacting together, and a team forced to investigate these two categories through entirely separate, disconnected tooling loses considerable diagnostic efficiency compared to one that can trace a single problematic request through its complete path, LLM calls and conventional application logic together, within one unified observability system built to represent both coherently.

How tracing underpins the broader practices of agent observability and evaluation

The detailed trajectory reconstruction covered in the discussion of agent observability, and the sampled, real-traffic evaluation covered in the discussion of ongoing agent evaluation, both ultimately draw their underlying raw data from exactly the same foundational trace records this discussion covers, agent observability organizing and visualizing that raw trace data specifically around an agent’s plan and action sequence, and evaluation sampling from that same trace data to select and review interactions against a defined quality rubric. Neither of these more higher-level practices could function without the underlying tracing infrastructure providing consistent, reliable, sufficiently detailed raw material for them to build on.

Recognizing this dependency explicitly should shape how a team prioritizes its observability investment, building solid, comprehensive tracing early, before investing heavily in the additional tooling and processes that sit on top of it, since a team that builds agent observability or evaluation on top of inconsistent or incomplete underlying tracing will find those higher-level investments delivering considerably less value than they otherwise would, constrained by gaps in the exact same foundational data those higher-level practices depend on to function well.

How to decide what to capture in a trace versus what to leave out

Capturing everything technically possible within every single span, every intermediate detail, every internal state variable, quickly produces trace data so voluminous and so cluttered with low-value detail that it becomes harder, not easier, to extract useful signal from, which means deciding what to capture is itself an important design decision rather than something to default toward maximal, indiscriminate capture without deliberate thought. The most useful traces capture the detail that supports the diagnostic and analytical questions a team expects to ask, prompts and completions, token counts, timing breakdowns, tool call parameters and results, while deliberately leaving out detail that’s technically available but rarely, if ever useful for any diagnostic purpose.

This decision also has to weigh the privacy and cost implications covered in the discussions of production monitoring and agent observability more broadly, since capturing more detail means capturing more potentially sensitive content and generates storage cost at scale, and a team should specifically, deliberately weigh a piece of additional captured detail’s diagnostic value against those costs, rather than capturing everything by default and only later, reactively, trying to figure out what to do with an unmanageable, unwieldy volume of low-value trace data it never deliberately decided it needed in the first place.

How tracing supports comparing the same request across different model or configuration variants

Beyond reconstructing what happened during one interaction, tracing infrastructure that captures consistent, structured data across many requests also supports a valuable comparative use case, running the exact same or equivalent input through two different model versions, two different prompt templates, or two different retrieval configurations, and directly comparing the resulting traces side by side to see exactly where and how the two variants diverge, not just in their final output but in the intermediate steps, timing, and token usage that led to that output. This comparative capability depends entirely on the underlying trace data being structured and consistent enough to support this kind of direct, side-by-side alignment between two separate traces.

Building tracing infrastructure with this comparative use case explicitly in mind means ensuring that spans are tagged with enough consistent metadata, which model version was used, which prompt template version, which configuration variant, that a team can later query and group traces along exactly these dimensions when running this kind of comparison, rather than only being able to inspect each trace as an isolated, standalone record with no way to systematically pull together and align traces sharing a variant of interest. A team that never designs its tracing with this comparative capability in mind typically discovers the gap only once it needs to run a before-and-after comparison and finds its trace data isn’t structured well enough to support that comparison efficiently.

How tracing handles errors, retries, and partial failures within a span

A model call or tool call doesn’t always simply succeed or cleanly fail, it can time out and get automatically retried, it can return a partial or malformed result that the calling code has to specifically detect and handle, or it can fail in a way that triggers a fallback to a different model or a different approach entirely, and useful tracing needs to represent this more nuanced reality explicitly within its span data rather than only recording a simple binary success-or-failure outcome that collapses all of this important detail into a single, oversimplified status flag.

Representing this well means capturing retry attempts as their linked, related spans rather than silently overwriting an earlier failed attempt with only the final, eventually successful one, since a trace that only shows the final successful attempt, with no record that three earlier attempts failed first, gives a considerably misleading impression of that call’s reliability and latency, hiding exactly the kind of retry-driven inefficiency covered in the broader discussion of production monitoring’s latency breakdown. A team whose tracing silently discards failed attempts in favor of only recording an eventual success loses the data needed to notice and diagnose this kind of hidden, retry-driven cost and latency problem before it grows into something considerably more significant.

How tracing scales in terms of engineering effort as an application’s architecture grows more complex

Tracing a single, simple LLM call is a relatively contained instrumentation task, but as an application’s architecture grows to include multiple chained calls, several distinct tools, retrieval steps, and eventually multiple collaborating agents, the engineering effort required to maintain consistent, complete tracing across that growing complexity grows right alongside it, and a team that treats tracing as a one-time setup task completed early in a project’s life, rather than as ongoing infrastructure that needs deliberate maintenance as the application itself evolves, typically finds its tracing coverage quietly degrading, developing gaps precisely in whichever newer, more recently added parts of the system nobody specifically remembered to instrument as carefully as the original, earlier parts.

Managing this growing complexity well means building tracing instrumentation into a team’s standard development practice, treating the addition of proper tracing to a new call, tool, or component as a standard, expected part of building that new piece in the first place, rather than as an optional afterthought to be added later if time permits, and periodically, deliberately auditing an application’s actual tracing coverage against its current architecture to catch any gaps that may have quietly accumulated as the system has grown and changed considerably since tracing was first originally set up.

How to validate that a tracing implementation is working correctly

A tracing implementation that appears to be working, producing traces that look reasonable when spot-checked occasionally, can still contain gaps or inaccuracies that only become apparent once a team needs to rely on that trace data for a high-stakes investigation, which is why validating tracing correctness deserves its deliberate, ongoing attention rather than being assumed correct simply because the tracing system is technically running without any visible errors of its own. Validation means periodically checking that traces for a known, deliberately constructed test interaction contain everything that interaction should have produced, no missing spans, no broken parent-child relationships, no silently dropped context propagation across a service boundary.

This kind of deliberate validation matters because tracing gaps tend to be silent failures, a dropped span or a broken context propagation doesn’t typically throw a visible error anywhere, it simply produces an incomplete trace that looks superficially normal until an investigation needs the exact piece of detail that happened to be missing, at which point the gap becomes a problem precisely when a team can least afford to discover it. A team that periodically, deliberately tests its tracing implementation against known scenarios catches this kind of silent degradation considerably before it costs a high-stakes investigation the detail it desperately needed.

Common mistakes teams make around LLM tracing

Several patterns recur often enough across teams building LLM tracing that naming them directly, explicitly is worth doing before they lead to a costly gap underneath the higher-level observability and evaluation practices that depend on this foundational layer.

1. Building higher-level observability and evaluation practices without first establishing solid, consistent tracing underneath them, resting those investments on an unreliable, shaky foundation.

2. Flattening a nested, chained sequence of calls into an undifferentiated list, losing the parent-child structure that makes a complex trace comprehensible.

3. Capturing span timing without the actual prompt and completion content, losing the substantive detail that supports content-level debugging and evaluation.

4. Building LLM tracing as an entirely separate, completely disconnected system from existing conventional application performance monitoring tooling already in place.

5. Getting trace context propagation wrong at a service boundary, producing traces that silently, mysteriously fragment exactly at that particular point.

6. Relying purely, entirely on manual instrumentation without any automatic baseline coverage, leaving gaps wherever a developer simply, honestly forgot to add explicit tracing code.

7. Ignoring standardized, well-established semantic conventions entirely, producing trace data that can’t be compared or aggregated using shared, common tooling across different systems.

8. Treating a streamed call’s total completion time as the only latency metric worth capturing, missing the distinct time-to-first-token signal that shapes actual perceived responsiveness.

9. Collapsing a single, individual call’s latency into one undifferentiated number rather than breaking out queueing, provider processing, and application-side handling as distinct, separate phases.

10. Choosing a trace storage backend without considering the actual balance between targeted single-trace lookup and broad, aggregate analytical query patterns a team needs.

11. Capturing everything technically possible by default, producing trace data so voluminous and cluttered that useful signal becomes considerably harder to extract rather than easier.

12. Never weighing a piece of additional captured detail’s diagnostic value against its privacy and storage cost implications before deciding to capture it.

13. Treating tracing as a one-time setup task rather than infrastructure that needs its ongoing maintenance as an application’s call structure continuously evolves.

14. Building agent observability or evaluation tooling that assumes trace data carries a level of consistency and completeness the underlying tracing implementation doesn’t reliably provide.

15. Underinvesting in tracing specifically because its value feels indirect, showing up primarily through the higher-level practices it supports rather than through any immediately visible benefit of its own.

16. Failing to tag spans with the consistent metadata needed to later group and compare traces across different model, prompt, or configuration variants.

17. Silently overwriting failed retry attempts with only the eventually successful one, hiding retry-driven latency and cost inefficiency the trace should have surfaced.

18. Treating tracing as a one-time setup task rather than ongoing infrastructure that needs deliberate maintenance as an application’s architecture grows considerably more complex over time.

19. Assuming a tracing implementation is working correctly simply because it produces traces that look reasonable on occasional spot-checks, without ever deliberately validating it against known scenarios.

What connects all nineteen of these mistakes is a single underlying pattern: treating LLM tracing as a secondary, supporting detail rather than as the foundational infrastructure layer that every more visible, higher-level observability and evaluation practice depends on to function well, and underinvesting in that foundation because its value is indirect, expressed through the practices built on top of it rather than through any immediately visible benefit of its own.

The deeper principle underneath all of this is that a team’s ability to understand and debug its AI system, at any level of sophistication, from a simple single-turn chatbot to a complex, multi-agent workflow, is fundamentally bounded by the quality and completeness of the underlying trace data that system really produces, and no amount of sophistication in the higher-level tools built on top of that data can ever compensate for gaps, inconsistencies, or missing detail at this foundational layer, which is exactly, precisely why LLM tracing deserves the exact same careful, deliberate investment a team gives to any other piece of infrastructure its most important, higher-level capabilities depend on to function well.