What is LLM observability?

Summary slides

Why LLM observability exists as its broad discipline

A deployed LLM system’s actual behavior is, by its nature, considerably harder to directly inspect than conventional software’s more predictable behavior, since a language model’s output emerges from a complex, largely opaque internal process rather than from explicit, cleanly traceable logic a developer deliberately wrote directly, and a system built around several chained calls, tool invocations, and retrieval steps compounds this opacity considerably further, each additional step adding another point where a team can lose visibility into what happened unless it has deliberately built the infrastructure needed to preserve that visibility. LLM observability exists specifically to counter this compounding opacity, building the deliberate, structured visibility that simply doesn’t ever emerge automatically just because a system happens to be running at all.

This need has grown directly, considerably alongside the field’s growing sophistication, a simple, single-call chatbot needs comparatively little dedicated observability infrastructure to understand well, while a complex agent system, chaining many calls, invoking tools, maintaining actual state across an extended, considerable trajectory, becomes functionally, practically impossible to understand or debug without dedicated observability infrastructure deliberately built to make its otherwise invisible internal behavior visible. The discipline’s growing importance tracks directly, quite closely with the broader field’s growing structural complexity over time, and shows no signs of that relationship reversing anytime soon.

What distinguishes observability from evaluation as parallel but complementary disciplines

Observability and evaluation, though closely related and often discussed together in the same breath answer different, distinct questions using different methods entirely. Observability asks what a system is doing right now, continuously, through structural instrumentation, traces, metrics, logs, that capture behavior exactly as it occurs in real time. Evaluation asks whether that behavior is good, through deliberate, sampled review against carefully defined quality criteria, a process that’s inherently more selective and more judgment-intensive than the continuous, structural capture observability itself performs. A system can have excellent observability capturing every single detail of what happened, while still entirely lacking any evaluation of whether what happened was truly good, and a different system can be evaluated carefully, thoroughly against a sampled set of interactions while still lacking the observability infrastructure needed to properly understand an incident whenever one eventually occurs in practice.

The two disciplines depend on each other in practice, evaluation needs observability’s trace data as its actual raw material to sample from and carefully review, and observability’s continuous signals benefit enormously from being interpreted through evaluation’s established sense of what truly constitutes good, acceptable behavior in practice. Treating either discipline as a complete substitute for the other one leaves a gap behind either way, comprehensive observability without evaluation tells a team everything that happened without ever judging whether any of it was really, truly good, while comprehensive, thorough evaluation without observability leaves a team unable to investigate the interactions its evaluation process happens to flag as concerning in the end.

The three pillars of observability adapted to LLM systems

Conventional software observability is often described through three foundational pillars, traces, metrics, and logs, and LLM observability adapts each of these three pillars directly to the structure of LLM-powered systems overall. Traces, covered in depth in the dedicated discussions of LLM tracing and agent tracing, capture the detailed, structured record of individual calls and, for agents, entire multi-step trajectories. Metrics capture aggregate, numerical signals, error rates, latency percentiles, cost per request, actual token consumption, that summarize behavior across many separate interactions rather than any single, individual one. Logs capture discrete, timestamped events, an error, a configuration change, a safety refusal, that don’t necessarily, always fit neatly into the more structured trace or metric formats but still carry diagnostic value nonetheless.

These three pillars work considerably best together rather than in complete isolation from each other, a metric showing an unusual latency spike tells a team something is wrong somewhere, a trace lets that same team drill directly into an affected interaction to understand why, and a log entry might reveal a configuration change that happened to coincide precisely with when that same metric first started shifting. A team that builds only one of these three pillars well, comprehensive traces but no aggregate metrics, or detailed metrics but no way to drill into a trace, ends up with gaps in its ability to understand and respond to what its system is doing.

How the sub-practices fit together as parts of this broader discipline

The many practices covered elsewhere in this discussion, LLM tracing providing the foundational span infrastructure, agent tracing extending that foundation to capture an agent’s particular execution structure, production monitoring watching operational health, cost monitoring and token monitoring tracking spend and technical resource consumption, agent observability providing the higher-level tools for using and analyzing trace data, each address a distinct part of the broader question LLM observability as a whole exists to answer, and none of them, taken alone, provides complete coverage.

Understanding how these pieces fit together prevents a common, costly mistake, treating strength in one of these more practices as though it settled the broader question observability as a whole is meant to answer. A team with excellent cost monitoring but no tracing infrastructure underneath it can track spend accurately while remaining unable to explain why an interaction cost what it did. A team with excellent agent observability tooling but inconsistent underlying tracing finds that tooling delivering considerably less value than it otherwise would, constrained by gaps in the exact same foundational data it depends on. Each piece is necessary, and no single piece is sufficient on its own.

How LLM observability differs from conventional software observability

Conventional software observability, well-established and mature, focuses on relatively predictable failure modes, a service crashes, a request times out, an error code gets returned, failure modes that tend to be binary and unambiguous. LLM observability inherits this conventional foundation but has to extend it considerably to cover failure modes conventional observability was never built to catch, a call that technically succeeds while returning a substantively degraded or hallucinated result, a model that behaves subtly differently after a silent, provider-side update, a token count that silently exceeds a context window and produces truncation rather than any visible error at all.

This extension means LLM observability can’t simply adopt conventional tooling wholesale, it has to build new instrumentation, new metrics, and new analytical approaches specifically suited to these LLM-failure modes, while still integrating cleanly with the conventional observability infrastructure most organizations already have in place for the rest of their application stack. A team that treats LLM observability as simply an application of conventional techniques to a new kind of system will miss the distinct failure modes this discussion’s more sub-practices were each built specifically to catch.

The role of dashboards and alerting across the whole observability stack

Raw trace, metric, and log data only becomes useful once it’s surfaced to the people who need to act on it, through dashboards that make current system health scannable at a glance, and alerting that proactively notifies a team when something has gone wrong rather than requiring someone to actively check a dashboard to notice. Building this surface layer well across an entire observability stack means applying the same layered design principle covered throughout this collection’s more discussions, a high-level summary view for quick, routine health checks, and progressively more granular drill-down views for the investigation a concerning signal warrants.

This surface layer also has to serve different audiences well, an engineer actively debugging an incident needs considerably more granular detail than a product leader checking on overall system health, and building distinct views tailored to each of these different audiences, rather than forcing everyone through one single, undifferentiated interface, is what makes an observability practice broadly useful across an entire organization rather than only useful to the engineers who built it.

How observability data flows from raw traces up to aggregate insight

A single trace captures what happened during one interaction, but the value of an observability practice compounds considerably once traces get aggregated into broader patterns, computing an average latency, an error rate, a cost-per-feature breakdown, from many individual traces rather than from any single one in isolation. This aggregation is what turns raw, granular trace data into the kind of high-level metric a dashboard can meaningfully display, and building this aggregation pipeline well means the underlying trace data has to be structured consistently enough, the same schema and the same field names applied reliably across every trace a system produces, for that aggregation to work correctly at scale.

This flow from raw trace to aggregate insight also has to work in reverse, letting a team start from a concerning aggregate metric and drill directly down to the traces that explain it, closing the loop between a high-level signal, something looks wrong, and the granular evidence needed to understand why. An observability practice that only supports aggregation in one direction, up from traces to metrics but never back down from a concerning metric to its underlying traces, leaves a team able to notice that something is wrong without any efficient path to understanding why.

How observability supports incident response end to end

A production incident moves through several distinct phases, detection, an alert fires or a dashboard shows a concerning trend, diagnosis, a team investigates to understand the root cause, and resolution, a team takes action to fix the problem and confirm the fix worked, and a well-built observability practice supports every one of these phases with the appropriate tooling, alerting and dashboards for detection, detailed traces and logs for diagnosis, and the ability to compare metrics before and after a fix for confirming resolution.

A team whose observability practice only supports one or two of these phases well finds its incident response correspondingly hampered, strong detection but weak diagnostic tooling leaves a team knowing something is wrong without being able to efficiently figure out why, while strong diagnostic tooling but weak detection leaves incidents going unnoticed until a frustrated user report eventually surfaces them long after they began. Building observability with this full incident-response lifecycle explicitly in mind, rather than optimizing for only one phase in isolation, produces a practice that practically helps a team through an incident from start to finish.

How observability practice matures as an organization scales

A small team running a single, relatively simple LLM-powered feature can often get by with fairly basic observability, some logging, a simple dashboard, manual review of occasional issues, but this basic approach stops scaling well once an organization’s systems grow more complex, more numerous, and more consequential, and the observability practice that was adequate at an earlier, smaller stage needs deliberate, ongoing investment to keep pace with that organizational growth rather than being treated as a one-time setup that’s simply left unchanged indefinitely.

This maturation typically follows a recognizable pattern, moving from ad hoc, manual investigation toward deliberately structured tracing and metrics, then toward cross-system correlation and automated alerting, and eventually toward the kind of proactive, pattern-based analysis that catches emerging problems before they’ve become serious incidents at all. A team that recognizes which stage of this maturation it’s currently at, and what specifically the next stage requires, can invest its limited observability effort considerably more effectively than one defaulting to whatever tooling happens to be easiest to adopt without regard for whether it matches the organization’s current needs.

The organizational and tooling ecosystem around LLM observability

A mature LLM observability practice typically draws on a mix of purpose-built LLM observability platforms, extensions to existing conventional application performance monitoring tooling, and custom, in-house instrumentation built specifically around a team’s particular needs, and choosing the right mix depends heavily on an organization’s existing tooling investments, its scale, and the gaps its particular systems need covered that off-the-shelf tooling doesn’t already address well. No single tool or platform currently covers every dimension of LLM observability equally well, which means most mature practices end up combining several distinct tools rather than relying on any single one exclusively.

This ecosystem is also still actively evolving, with new tools and new standards, like the OpenTelemetry semantic conventions covered in the discussion of LLM tracing, continuing to emerge and mature considerably as the broader field itself continues to grow, which means a team’s tooling choices deserve periodic revisiting rather than being treated as a permanent, unchanging decision made once early on and never reconsidered as better options become available over time.

How observability connects back into system and prompt improvement

Observability’s value extends well beyond passive monitoring and reactive incident response, it also provides the concrete, empirical evidence a team needs to improve its system deliberately, revealing which prompts produce unusually long or unusually expensive responses, which tool calls fail most often, which categories of request take an agent an unusually large number of steps to complete. This evidence turns observability into an active tool for ongoing system improvement, not merely a passive safety net that only ever gets consulted once something has already gone visibly wrong.

Building this connection deliberately means routing observability insights directly to the people and processes responsible for prompt engineering, tool design, and broader system architecture, rather than letting valuable observability data sit unused in a dashboard nobody outside the immediate observability team ever reviews. A team that builds this connection well finds its system improving continuously, informed by empirical evidence about how it’s behaving, rather than improving only sporadically, informed mainly by intuition or by the occasional serious incident that happens to force attention onto a particular problem.

Common pitfalls in building an LLM observability practice

Beyond the mistakes already covered in each sub-practice’s dedicated discussion, building LLM observability as a broader, coherent whole carries its distinct risks worth naming directly, building each sub-practice, tracing, monitoring, cost tracking, in isolation without ever connecting them into one coherent, integrated whole, investing heavily in sophisticated tooling while neglecting the organizational processes, ownership, escalation paths, that determine whether that tooling gets used effectively, and treating observability as a purely technical concern disconnected from the broader evaluation and quality practices it should be feeding into and drawing from.

Avoiding these broader pitfalls means treating LLM observability from the start as a coherent, integrated discipline rather than a loose collection of disconnected point solutions each solving one narrow problem in isolation, with deliberate attention paid not just to the technical instrumentation itself but to the organizational structure, ownership, process, and cross-team connection that determines whether all that technical investment translates into practical value for the organization building and operating the system.

How to choose or build LLM observability tooling

Deciding whether to adopt an existing, purpose-built observability platform or to build custom, in-house tooling depends on weighing trade-offs, an existing platform typically offers faster initial setup and ongoing maintenance handled by someone else, while custom tooling offers considerably more precise control over exactly what gets captured and how it gets analyzed, tailored specifically to an organization’s particular systems and its particular needs in a way a general-purpose platform, built to serve many different organizations at once, inevitably can’t match quite as precisely.

The most effective approach for many organizations blends both, adopting existing tooling for the foundational, broadly applicable layers, general tracing infrastructure, standard metric dashboards, while building custom instrumentation specifically for the particular, organization-specific needs a general-purpose platform doesn’t already address well, an agent architecture’s particular execution model, a compliance requirement around how sensitive trace data has to be handled. A team should specifically resist the temptation to build everything custom simply because an existing platform doesn’t perfectly match every single one of its needs, since the ongoing maintenance cost of fully custom tooling is often considerably higher than it initially appears once that tooling has to keep pace with a system’s continuing evolution over time.

Where LLM observability is heading

As LLM-powered systems grow more autonomous, more agentic, and more deeply embedded in consequential real-world decisions, LLM observability is visibly evolving to keep pace, moving from simple call-level tracing toward the rich, structural agent tracing covered in this discussion, from purely reactive, after-the-fact investigation toward proactive, pattern-based detection that catches emerging problems before they’ve become serious, and from siloed, single-team tooling toward integrated, organization-wide practices that connect observability directly into evaluation, incident response, and ongoing system improvement as one coherent, connected whole rather than several separate, disconnected efforts.

The organizations and practitioners best positioned for whatever comes next in this continuing evolution are the ones building observability practices flexible enough to keep adapting alongside their systems’ continuing growth in capability and complexity, rather than treating today’s tooling and today’s practices as a permanent, finished solution, since the systems being observed will keep changing considerably, and an observability practice that stops evolving alongside them will, sooner or later, find itself blind to precisely the kind of new failure mode its static, unchanging approach was never built to catch in the first place.

How observability should treat the cost of observability itself

Building comprehensive observability, detailed traces, granular metrics, extensive logging, carries a non-trivial cost of its own in storage, in compute overhead for the instrumentation itself, and in the ongoing engineering effort needed to build and maintain it, and a team building an observability practice has to weigh this cost against the diagnostic and operational value that additional observability investment delivers, rather than treating more observability as unambiguously, unconditionally better regardless of what it costs to build and sustain. A system instrumented so thoroughly that the observability overhead itself meaningfully degrades the underlying system’s performance has traded one problem for another, arguably worse one.

This weighing decision means a mature observability practice applies the same deliberate, tiered thinking covered throughout this discussion’s more sub-practices, comprehensive, detailed capture for the categories of interaction that carry the most diagnostic or business value, and more selective, lighter-weight capture for routine, lower-stakes interactions where the marginal value of additional detail doesn’t justify its ongoing cost. A team that never makes this trade-off deliberately, either capturing everything indiscriminately or under-investing broadly to avoid the cost entirely, ends up with an observability practice considerably less effective than one built around deliberate, cost-aware prioritization.

How observability should account for privacy and compliance requirements across its entire stack

The privacy considerations covered specifically for trace data in the discussions of LLM tracing and agent observability extend across an entire observability practice, since metrics and logs, not just detailed traces, can also inadvertently capture or reveal sensitive information, an aggregate metric broken down by a small enough user segment can itself leak information about an individual, a log entry capturing an error message might inadvertently include a fragment of sensitive user content that triggered that error in the first place. Building compliant observability means applying deliberate privacy discipline consistently across every layer of the stack, not just the most obviously detailed layer, trace data, while treating metrics and logs as though they carried no privacy risk simply because they appear more aggregated or more abstract on their surface.

This consistent discipline matters considerably more as an organization’s systems handle increasingly sensitive categories of user data, and building it in from the start, rather than retrofitting it reactively once a privacy or compliance concern has already been raised, is considerably less costly and considerably more reliable than the alternative. A team building LLM observability specifically for a regulated industry, healthcare, finance, legal services, should treat this privacy discipline as a foundational design requirement shaping every layer of its observability stack from the very beginning, rather than as an afterthought addressed only once the rest of the practice has already been built without it.

How to build cross-team ownership of an observability practice

LLM observability, done well, ends up mattering to considerably more people than the engineering team that originally built the underlying instrumentation, a product team wants to understand feature usage patterns, a finance team wants accurate cost attribution, a safety or compliance team wants visibility into refusal and boundary-testing behavior, and a mature observability practice has to serve all of these different stakeholders well rather than being built narrowly around only the original engineering team’s needs and then treated as though it were complete. Building this broader, cross-team ownership deliberately, rather than letting observability remain the exclusive domain of whichever team happened to build it first, is what turns a technically excellent observability system into an organization-wide asset rather than a narrow, single-team tool most of the organization never benefits from or even knows exists.

This cross-team ownership means specifically soliciting input from these other stakeholders when designing an observability practice’s dashboards, metrics, and alerting, rather than assuming the original engineering team already knows exactly what every other team needs without ever asking them directly, and it means establishing clear processes for how these other teams can access and act on the observability data relevant to their concerns, rather than leaving that data technically accessible but practically locked behind tooling and interfaces only the original engineering team knows how to use effectively. An observability practice that never achieves this broader ownership tends to remain a valuable but narrowly used tool, delivering only a fraction of the organization-wide value it could otherwise provide.

How observability practice should handle the transition when a system migrates between underlying models or providers

A migration from one underlying model or provider to another, whether driven by cost, capability, or simple provider availability, is exactly the kind of significant system change where observability’s value becomes most immediately, concretely apparent, providing the before-and-after comparison data needed to verify that a migration hasn’t introduced an unexpected regression in latency, cost, quality, or safety behavior, extending directly the kind of bracketed before-and-after evaluation covered in the broader discussion of ongoing agent evaluation into the operational domain observability itself covers.

Building effective migration support into an observability practice means specifically preserving the ability to compare metrics and traces across the exact moment a migration occurred, tagging data clearly with which model or provider version generated it, so a team can directly, confidently attribute any observed change in behavior to the migration itself rather than to some other, entirely unrelated factor that happened to shift at roughly the same time by pure coincidence. A team whose observability practice doesn’t specifically support this kind of clean, clear before-and-after comparison across a migration boundary finds itself considerably less able to confirm a migration went well, or to quickly identify and diagnose specifically what went wrong if it didn’t.

Common mistakes teams make around LLM observability

Several patterns recur often enough across organizations building LLM observability as a broad, coherent practice that naming them directly, explicitly is worth doing before they lead to a costly gap in an organization’s overall ability to understand and respond to how its deployed systems are behaving in practice.

1. Building tracing, monitoring, and cost tracking as entirely, completely separate, disconnected efforts rather than as integrated parts of one coherent observability practice.

2. Treating observability and evaluation as entirely interchangeable, missing the distinct question each one is built to answer and the way each depends on the other.

3. Investing heavily in sophisticated technical tooling while neglecting the organizational ownership and escalation process that determines whether that tooling gets used effectively.

4. Building only aggregate metrics without the underlying trace detail needed to drill down and explain a concerning metric once one is noticed.

5. Treating conventional software observability techniques as directly, entirely sufficient, missing the distinct LLM-failure modes this discussion’s sub-practices were each specifically built to catch.

6. Building dashboards that only serve one audience well, either overwhelming non-technical stakeholders or under-serving the engineers who need granular, technical detail.

7. Supporting only one single phase of the incident-response lifecycle well, strong detection without diagnostic depth, or strong diagnosis without adequate initial detection.

8. Treating an observability practice adequate for an earlier, considerably smaller organizational scale as though it remains equally adequate without any deliberate, ongoing reinvestment as the organization considerably grows.

9. Building fully custom observability tooling entirely by default rather than weighing it against existing, purpose-built platforms that already cover much of the same essential ground well.

10. Letting observability insights sit entirely unused in a dashboard nobody outside the immediate observability team ever reviews, missing the connection back into prompt and system improvement.

11. Adopting observability tooling only once and never revisiting that choice as the broader tooling ecosystem and relevant standards continue to evolve and mature.

12. Treating each sub-practice, tracing, agent observability, cost monitoring, as though strength in one settled the broader question observability as a whole is meant to answer.

13. Building observability infrastructure without ever validating that data reliably flows correctly both up from traces to aggregate metrics and back down from a concerning metric to its underlying traces.

14. Assuming observability infrastructure adequate for a single-call chatbot remains equally adequate once a system grows into a complex, multi-step agentic architecture.

15. Treating LLM observability as a purely technical, engineering-owned concern entirely disconnected from the broader evaluation and quality practices it should be feeding into and drawing from.

16. Treating additional observability instrumentation as entirely, unconditionally beneficial without weighing its storage, performance, and ongoing maintenance cost against its actual diagnostic value.

17. Treating metrics and logs as inherently, entirely privacy-safe simply because they appear more aggregated than detailed traces, missing how they can still inadvertently leak sensitive information.

18. Building observability narrowly, entirely around the original engineering team’s needs, leaving interested stakeholders in product, finance, and compliance without practical access.

19. Failing to preserve clean before-and-after comparison across a model or provider migration, leaving a team unable to confidently attribute an observed change to the migration itself.

What connects all nineteen of these mistakes is a single underlying pattern: treating LLM observability as a collection of disconnected point solutions rather than as the coherent, foundational discipline it is, one that every more practice covered throughout this collection, tracing, monitoring, cost tracking, agent observability, evaluation itself, ultimately depends on, and underinvesting in the deliberate integration work that turns those separate pieces into one coherent whole considerably more valuable than the sum of its individual, disconnected parts.

The deeper principle underneath all of this is that a team’s ability to build, operate, and continuously improve a trustworthy AI system rests fundamentally on its ability to see what that system is doing, and this visibility doesn’t emerge automatically just because a system happens to be running successfully, it has to be deliberately built, layer by layer, from foundational tracing up through operational monitoring to organization-wide insight, weighed carefully against its cost, protected by privacy discipline, and owned broadly across every team that depends on it, and only an organization that treats this entire observability discipline with the same seriousness it gives to building the system itself can claim to truly, honestly understand, at any moment, what its deployed AI system is really doing and why.