What is production AI monitoring?

Summary slides

Why production AI monitoring exists as its distinct practice from evaluation

Evaluation asks whether an AI system’s actual output quality meets the bar that matters, sampling interactions and scoring them against carefully defined rubrics, a process that’s inherently somewhat slower and considerably more deliberate since quality judgment, whether rendered by a careful human reviewer or a calibrated automated grader, takes real time to produce a reliable, trustworthy result. Monitoring asks a different, more immediate question: is the system running correctly right now, is it responding within acceptable time, is it failing at an unusual rate, is it costing more than expected, questions that can and should be answered continuously, in near real time, rather than through periodic sampling and review.

This distinction matters because the two practices catch different categories of problem at different speeds. A sudden spike in error rate caused by an upstream API outage needs to be caught and responded to within minutes, a considerably faster timescale evaluation, built around careful sampling and deliberate review, was never designed to operate at all. A gradual decline in output quality caused by subtle prompt drift or an unnoticed, gradual model behavior change is exactly the kind of problem monitoring’s operational metrics simply won’t directly reveal on their own, since a system can be running perfectly from an operational standpoint, fast, reliable, within cost budget, while its actual output quality has quietly degraded in ways only evaluation would catch. Production AI monitoring and evaluation are truly complementary practices precisely because each one catches problems the other one structurally, fundamentally cannot ever catch on its own.

What operational signals production AI monitoring tracks

Latency tracks how long a system takes to respond, both the typical, ordinary response time and, critically, the tail latency, the slowest actual requests, since a system with excellent average latency but a long tail of considerably slow outlier requests can still frustrate a meaningful fraction of users even while its headline average number continues to look perfectly healthy on the surface. Error rate tracks how often a request truly fails outright, whether due to an upstream API failure, a timeout, an unexpected input the system couldn’t handle, or an internal error somewhere in the application logic surrounding the model itself.

Throughput tracks how much actual volume the system is handling, requests per second or per minute, a signal that matters both for capacity planning and for detecting unusual traffic patterns that might indicate anything from a legitimate surge in usage to a misconfigured client sending unintended, repeated requests over and over. Cost tracks the actual spend the system is generating, token usage, API calls, compute, broken down specifically enough to catch a cost spike concentrated in one particular feature or one particular category of request rather than only ever visible as a vague, undifferentiated, blended total number. Together, these operational signals give a team the real-time pulse of whether a system is truly healthy at a purely infrastructural level, entirely independent of whether its actual outputs are good ones.

How monitoring an AI system differs from monitoring conventional software

Conventional software monitoring rests on a relatively stable, well-understood set of failure modes, a server crashes, a database connection times out, a dependency becomes unavailable, failure modes that tend to be binary and unambiguous, a request either succeeded or it visibly, obviously failed. AI systems introduce new categories of operational concern conventional monitoring was never built to track: a request to an underlying model provider can succeed technically, returning a response within normal latency and without any error code, while that response itself is degraded, truncated, or behaviorally different from what the system was built around, a failure mode invisible to monitoring built purely around traditional success-or-failure signals.

This means AI system monitoring has to track dimensions conventional monitoring typically ignores entirely: token-level metrics specific to how a language model generates its output, and distributional signals tracking whether the actual shape of a model’s responses, their length, their structure, their apparent confidence, has shifted in ways that might indicate an underlying model or provider-side change, even when every individual request still technically, nominally succeeded. A monitoring practice built purely on the conventional playbook, tracking only basic uptime and basic error codes, will miss an entire category of AI-operational problem that manifests as a technically successful but substantively different response.

How to monitor for model output drift specifically

Model output drift refers to a gradual or sudden shift in a model’s actual behavior, its typical response length, its tone, its accuracy on the kinds of requests a system regularly handles, that occurs entirely without any deliberate, intentional change on the deploying team’s side, most commonly triggered by an upstream provider silently updating the underlying model version being served, or by a shift in the actual distribution of incoming requests the system is facing day to day. Detecting this kind of drift requires tracking statistical summaries of actual model output over time, average response length, the distribution of structural patterns in generated responses, and flagging a meaningful, sustained deviation from an established baseline rather than reacting to any single unusual response in isolation.

This kind of drift detection sits in a useful middle ground between pure operational monitoring and full evaluation, it doesn’t require the kind of careful, qualitative judgment full evaluation demands, but it goes beyond simple uptime and latency tracking to catch a category of problem, a model quietly behaving differently, that neither pure operational monitoring nor periodic evaluation sampling alone would catch quickly. A team that specifically, deliberately instruments this kind of drift detection gains an early warning signal considerably faster than the one a periodic evaluation cycle would eventually provide, often catching a meaningful behavioral shift within hours rather than waiting for the next scheduled evaluation review to eventually surface it.

How to monitor cost and token usage at production scale

Cost monitoring for an AI system needs considerably finer, more detailed granularity than a single, undifferentiated aggregate spend number provides, since a useful cost monitoring practice breaks spend down by feature, by user segment, by the model or provider being used, so that a cost anomaly can truly be traced back to its source rather than only ever showing up as an unexplained, mysterious jump in an undifferentiated total. A sudden cost spike concentrated in one particular feature points directly at an actionable place to investigate, while the same spike buried inside a single blended total gives a team considerably less to work with when trying to understand what changed.

This granular tracking also matters for catching an easy-to-miss failure mode: a bug that causes a system to retry a failed request excessively, or a prompt that’s grown unintentionally longer over successive iterations without anyone specifically noticing, can each drive cost up considerably without producing any visible functional problem at all, since the system still technically, nominally works correctly from a user’s perspective even as its actual cost efficiency quietly, steadily degrades over time. Cost monitoring built with this kind of granularity catches exactly this category of quiet, functionally invisible degradation that neither error-rate monitoring nor user-facing evaluation would ever directly reveal on its own.

How alerting works for AI systems and what thresholds make sense

An alerting system only delivers actual value if its thresholds are calibrated well enough to catch problems without also generating so much noise, so many false, misleading alarms, that the team receiving those alerts eventually starts ignoring them entirely, a well-documented, widely recognized failure mode called alert fatigue that undermines the entire purpose of having alerting in the first place at all. Setting useful thresholds for an AI system requires understanding that its operational metrics carry considerably more natural variance than conventional software’s typically do, latency in particular can vary meaningfully across different types of request depending on how much the underlying model has to generate, which means a threshold calibrated too tightly around a simple, unconditional average will trigger constantly on entirely normal, expected variation rather than any problem.

The most effective alerting for AI systems is built around sustained deviation from an established baseline rather than any single data point crossing a fixed, arbitrary threshold, alerting when error rate has stayed elevated for several consecutive minutes rather than firing on one isolated failed request, and alerting when a cost or latency metric has drifted meaningfully away from its recent historical pattern rather than simply exceeding some fixed number chosen without reference to what normal variation looks like for that system. This baseline-relative approach produces alerts that are considerably more likely to represent a problem worth a team’s immediate attention.

How to monitor for latency and its AI-related causes

Latency in an AI system has several distinct potential causes that conventional software latency monitoring wouldn’t naturally, automatically distinguish between: the underlying model provider’s response time, which a deploying team has limited direct control over but still needs clear visibility into; queueing delay, if the system is handling more concurrent requests than its current capacity comfortably supports; and retry behavior, since a system that automatically retries a failed or slow request can produce end-to-end latency considerably worse than any single attempt’s latency would suggest, a pattern invisible to monitoring that only tracks the final, eventually successful response time rather than the full sequence of actual attempts that led to it.

Useful latency monitoring breaks these root causes apart quite explicitly, tracking model provider response time, internal queueing delay, and retry count as separate, distinct signals rather than only reporting one single, blended end-to-end latency number, since a team trying to improve latency needs to know specifically which of these several distinct causes is driving a problem before it can decide what to fix. A blended latency number that has degraded tells a team something is wrong without telling them where, while the same degradation broken down by cause points directly at whether the problem sits with an upstream provider, with the system’s current capacity, or with an overly aggressive retry policy amplifying a smaller underlying issue into a considerably larger one.

How logging and observability work specifically for AI systems

Debugging a conventional software failure typically means tracing a request through a sequence of function calls and service boundaries, and AI system observability extends this same basic idea specifically to cover the additional layers an AI system introduces, the exact prompt sent to a model, the full sequence of tool calls an agent made along the way, the retrieved passages a RAG system’s retrieval component surfaced, each of which needs to be logged and traceable if a team is ever going to understand why an interaction produced the particular output it did.

This tracing capability matters enormously once a problem is detected, whether through monitoring’s operational alerts or through evaluation surfacing a quality issue, since diagnosing the root cause of either kind of problem requires being able to reconstruct exactly what happened for an affected interaction, not just knowing in the abstract that some category of interaction is currently performing poorly. A team without this kind of detailed tracing in place can detect that something is wrong considerably faster than it can understand why, leaving root-cause diagnosis stuck at the level of educated guessing rather than grounded, evidence-based investigation.

How to monitor for safety and abuse patterns in production

Beyond ordinary operational health, production AI monitoring needs its dedicated attention to safety-relevant patterns that only become visible once a system is facing live traffic, a sudden concentration of requests attempting to elicit harmful content, a pattern of requests that look like a coordinated attempt to probe the system’s boundaries or extract information it shouldn’t reveal, or usage that suggests the system is being deployed for a purpose meaningfully different from what it was built and evaluated for. This category of monitoring sits closer to security monitoring than to ordinary operational health tracking, but it belongs within a comprehensive production AI monitoring practice precisely because these patterns can only be detected through the same kind of continuous, real-time observation the rest of monitoring already provides.

Building this monitoring well means tracking not just whether individual requests get correctly refused when they should be, a signal closer to safety evaluation, but the aggregate pattern of refusal and boundary-testing behavior over time, since a sudden spike in this pattern can indicate an emerging threat worth investigating even when every single individual request is technically being handled correctly on its own. A team that only tracks whether individual requests are handled correctly, without also tracking the aggregate pattern those requests form over time, can miss a coordinated attempt playing out gradually across many individually unremarkable interactions.

How incident response works specifically for AI system failures

When production monitoring surfaces an incident, an actual spike in error rate, a latency degradation, a cost anomaly, effective incident response for an AI system follows the same basic discipline conventional software incident response relies on, quickly establishing the actual scope and severity of impact, identifying a likely root cause, and taking whatever immediate mitigating action stabilizes the system while a permanent fix gets developed, but with a few AI-wrinkles worth understanding directly. An incident traced to an upstream model provider’s outage or silent model update often can’t be directly fixed by the deploying team at all, which means AI system incident response frequently needs a playbook for gracefully degrading, falling back to a different model or provider, returning a clear error rather than a degraded response, rather than the kind of direct fix conventional incident response typically aims for.

This means building incident response readiness for an AI system means having that fallback and degradation logic built and tested well before an incident ever occurs, not improvised under real time pressure once a provider-side outage has already started affecting users. A team that has never tested its fallback path before an incident occurs often discovers, in the middle of that incident, that the fallback path itself has its undiscovered problems, compounding rather than mitigating the original issue.

How monitoring dashboards should be designed for different audiences

A dashboard built for an on-call engineer actively responding to a real-time incident needs to surface exactly the granular signals, latency broken down by cause, error rate by endpoint, that let that engineer quickly diagnose what’s happening right now, while a dashboard built for a product leader checking on overall system health needs a considerably more condensed, higher-level view, is the system generally healthy, is cost trending in an expected direction, without the granular detail that would overwhelm someone who isn’t actively debugging a technical problem. Building a single, undifferentiated dashboard meant to serve both audiences equally well tends to serve neither one particularly effectively.

The most effective monitoring practices build distinct views for these different audiences, a detailed, granular operational view for the engineers who need to diagnose and fix problems, and a considerably more condensed, trend-focused summary for stakeholders who need to understand overall system health without wading through the underlying operational detail that view was never meant to communicate to them. A team that skips this differentiation and forces every audience through the same dashboard risks either overwhelming non-technical stakeholders with detail they can’t act on, or under-serving the engineers who need that detail to do their work.

How monitoring connects to and complements ongoing evaluation

Monitoring and evaluation, though distinct practices operating at different speeds and covering different questions, work best as a coordinated pair rather than as entirely separate, disconnected efforts. Monitoring’s real-time operational signals can trigger a targeted, focused evaluation review, a sudden shift in response length flagged by drift monitoring is exactly the kind of signal that should prompt an immediate, deliberate evaluation review of recent output quality, rather than waiting for that shift to eventually surface through a routine, scheduled evaluation cycle on its own.

Conversely, evaluation findings should inform what monitoring specifically tracks going forward, a quality problem evaluation discovers in a particular category of request should prompt monitoring to add a dedicated, ongoing operational signal specifically covering that category, closing the loop so that a problem evaluation had to discover manually once gets caught considerably faster, through real-time monitoring, the next time it starts to recur. Teams that build this bidirectional connection between monitoring and evaluation, rather than running the two practices in isolation from each other, catch both operational and quality problems considerably faster than either practice could manage entirely on its own.

How monitoring handles multi-provider or multi-model routing setups

Many production AI systems don’t rely on a single model or provider at all, routing requests across several different models based on cost, capability, or availability, sometimes falling back automatically from a primary provider to a secondary one when the primary is degraded or unavailable, and this routing complexity introduces its distinct monitoring requirements beyond what a single-model system would need. Monitoring has to track operational health and quality signals separately for each model and provider in the routing mix, since a routing setup that blends metrics across several different underlying models into one undifferentiated aggregate number can mask a problem specifically concentrated in just one of those models while the blended aggregate still looks perfectly acceptable.

This per-provider granularity matters directly for catching a common failure pattern in routed systems: a secondary, fallback provider that’s rarely used under normal conditions can silently develop its operational problems, a model deprecation, a configuration drift, that goes completely undetected until the exact moment that fallback path is needed during a primary-provider incident, precisely the worst possible time to discover a fallback path was never working correctly. Monitoring that specifically, periodically exercises and checks the health of rarely used fallback paths, rather than only monitoring whatever path is currently handling live traffic, catches this failure mode before it compounds an incident.

How monitoring evolves as usage and system scale grow

A monitoring setup adequate for a system handling a modest volume of requests can become considerably less adequate as that same system scales to handle meaningfully more traffic, both because the underlying infrastructure itself faces new operational challenges at greater scale, and because the sheer volume of monitoring data itself can grow to a point where naive approaches to storing, querying, and alerting on it stop working efficiently. A monitoring practice that was never revisited as a system scaled can quietly degrade in its usefulness, generating queries too slow to support real-time incident response, or accumulating so much raw data that meaningful signal gets buried under sheer, undifferentiated volume.

This means monitoring itself, not just the system it watches, needs periodic review as usage grows, checking whether alerting thresholds calibrated for an earlier, smaller scale of traffic are still meaningful at current volume, and whether the underlying monitoring infrastructure itself can still support the real-time diagnostic queries an active incident demands. A team that treats its monitoring setup as a fixed, one-time investment rather than something requiring its ongoing maintenance risks discovering its monitoring has quietly become inadequate exactly when a high-stakes incident, at greater scale than the monitoring was originally built around, makes that inadequacy most costly.

How monitoring should treat user-facing feedback signals alongside pure operational metrics

Beyond the purely infrastructural signals covered so far, users often generate their own direct and indirect feedback about a system’s quality, an explicit thumbs-down on a response, a rephrased follow-up suggesting the first answer didn’t help, an abandoned conversation partway through, and these user-generated signals occupy a useful middle ground between pure operational monitoring and deliberate, sampled evaluation, since they’re generated continuously by usage at scale the way operational metrics are, while carrying information about actual output quality the way evaluation does. A monitoring practice that ignores these signals entirely, tracking only infrastructural health, is leaving a considerable amount of continuously available quality signal completely unused.

Building this into a monitoring practice means treating a sustained shift in these user-feedback signals, a rising rate of explicit negative feedback, a growing rate of conversation abandonment, with the same urgency as an operational anomaly, triggering the same kind of immediate investigation a latency or error-rate spike would trigger, rather than letting these signals sit passively in a database waiting for someone to eventually notice them during a routine, periodic review. A team that treats user feedback as a real-time monitoring signal, rather than as raw material only ever reviewed later during a separate, deliberate evaluation cycle, catches a meaningful category of quality problem considerably faster than one that waits for evaluation alone to surface it.

How monitoring data itself needs to be retained, aggregated, and eventually discarded

Production AI monitoring generates a large, continuous volume of data, full request logs, detailed traces, granular per-request metrics, and a team has to make deliberate decisions about how long to retain that data at full granularity versus when to aggregate it into coarser summaries or discard it entirely, decisions that carry trade-offs between diagnostic usefulness storage cost, and, for any data that touches user content privacy obligations that constrain how long that content can be retained in the first place. Retaining everything at full granularity indefinitely is rarely the right default, since the cost of storing it eventually outweighs the diminishing value of being able to inspect an increasingly old interaction in full, granular detail.

A well-designed retention policy typically keeps full, granular detail for a useful recent window, long enough to support incident investigation and root-cause diagnosis for anything that happened recently, then progressively aggregates older data into coarser trend summaries that remain useful for tracking longer-term patterns without the storage cost or privacy exposure of retaining every individual raw interaction indefinitely. A team that never designs this retention policy deliberately, either keeping everything forever or discarding data too aggressively, ends up either accumulating unsustainable storage cost and privacy risk, or losing the diagnostic detail an incident investigation several weeks after the fact would need.

How monitoring ownership and on-call responsibility should be structured

Monitoring infrastructure that nobody is specifically responsible for watching and acting on tends to degrade into passive instrumentation that technically exists without serving its purpose, alerts firing into a channel nobody actively monitors, dashboards nobody regularly checks until well after a problem has already caused meaningful harm. Building effective production AI monitoring means pairing the technical infrastructure with explicit ownership, a defined on-call rotation with clear responsibility for responding to alerts, and a clear escalation path for incidents that exceed what the immediate on-call responder can resolve on their own.

This ownership question becomes more complicated for AI systems specifically because some categories of incident, a provider-side model update, an upstream API outage, sit partly or entirely outside what the deploying team itself directly controls, which means on-call responsibility for an AI system often needs an explicit, documented boundary distinguishing what the on-call responder can directly fix versus what requires escalating to, or simply waiting on, an external party, a distinction that’s considerably less common in conventional software on-call structures where the team on call typically owns the full, actual stack it’s responsible for. A team that never documents this boundary discovers it awkwardly, under real time pressure, in the middle of an incident that turns out to require an external fix nobody had planned for in advance.

Common mistakes teams make around production AI monitoring

Several patterns recur often enough across teams operating AI systems in production that naming them directly, explicitly is worth doing before they lead to a costly incident that monitoring should have caught considerably sooner than it did.

1. Relying purely, entirely on conventional software monitoring, uptime and basic error codes, missing the AI-failure modes where a request succeeds technically while its actual output has quietly degraded.

2. Tracking only average latency without any visibility into tail latency, missing the meaningful fraction of users experiencing considerably worse response times than the average suggests.

3. Setting alert thresholds around fixed, arbitrary numbers rather than deviation from an established baseline, producing either constant false alarms or missed problems.

4. Tracking cost only as a single, blended total, losing the granularity needed to trace a cost anomaly back to its actionable source.

5. Building a single, undifferentiated dashboard meant to serve both on-call engineers and non-technical stakeholders equally well, and serving neither one particularly effectively as an actual result.

6. Monitoring only the currently active model or provider in a routed system, leaving rarely used fallback paths with essentially no operational visibility at all whatsoever.

7. Treating monitoring and evaluation as entirely, completely separate, disconnected practices rather than building the feedback loop where each one informs and strengthens the other.

8. Blending end-to-end latency into one undifferentiated, single number rather than separating out model provider time, queueing delay, and retry behavior as distinct, separate signals.

9. Building incident response readiness without ever testing fallback and degradation logic before an incident forces that path into unplanned use.

10. Tracking only whether individual requests are handled correctly, missing the aggregate, broader pattern that reveals a coordinated safety-relevant threat playing out gradually over time.

11. Skipping detailed, request-level tracing and observability, leaving a team able to detect that something is wrong considerably faster than it can truly understand why.

12. Letting a monitoring setup calibrated for an earlier, considerably smaller scale of traffic go unrevisited as usage grows considerably, quietly degrading its usefulness.

13. Treating drift detection as entirely unnecessary because individual requests still technically succeed, missing gradual behavioral shifts that operational monitoring alone was never built to catch.

14. Ignoring provider-side model updates as a source of operational risk, assuming an upstream model’s behavior stays fixed unless the deploying team itself changes something.

15. Building monitoring infrastructure without periodically checking whether it can support the real-time diagnostic queries an active incident demands.

16. Treating user-generated feedback signals as raw material for later, separate evaluation review rather than as a real-time monitoring signal worth immediate investigation.

17. Retaining every single request log and trace indefinitely at full granularity, accumulating unsustainable storage cost and privacy exposure rather than designing a deliberate retention policy.

18. Leaving monitoring alerts without any explicit ownership or on-call structure, letting infrastructure exist passively without anyone responsible for acting on what it surfaces.

What connects all eighteen of these mistakes is a single underlying pattern: treating production AI monitoring as though it were simply conventional software monitoring with a different label, when it requires its distinct signals, drift, per-provider granularity, aggregate safety patterns, request-level tracing, built specifically around the ways an AI system can fail that conventional monitoring was never designed to catch in the first place.

The deeper principle underneath all of this is that an AI system’s operational health and its output quality are separate, distinct properties that can diverge from each other in either direction, a system can run perfectly at an operational level while its actual quality quietly, invisibly degrades, and a system can experience an operational incident while its output quality, for whatever traffic still gets through, remains perfectly fine, and only a monitoring practice built specifically to track both dimensions, in close coordination with the evaluation practice covering quality more deeply, and with explicit ownership behind it, can give a team the complete picture of how its deployed system is truly performing at any moment in real time.