What is LLM reliability?

Summary slides

Why LLM reliability means something different from conventional software reliability

Conventional software reliability rests on a foundation that’s easy to take for granted until it’s suddenly missing: given the same input and the same code, a conventional program produces the same output every time, and reliability engineering in that world is mostly about keeping the system available and keeping that deterministic behavior intact under load, under failure, and over time. An LLM breaks this foundation at its very root, because the same prompt sent twice to the same model can produce two different responses, which means the entire vocabulary of conventional reliability, uptime, error rate, mean time between failures, doesn’t map cleanly onto the actual thing that makes an LLM-based system unreliable in practice.

The gap becomes obvious once you notice that an LLM can be perfectly “available,” responding promptly to every request with no timeouts and no errors, while still being deeply unreliable in the sense that matters to whoever’s using the system: producing answers that are subtly wrong, inconsistently phrased, confidently overstated, or simply different in substance from one call to the next on the same input. None of that shows up in a conventional uptime dashboard, because nothing about it is a conventional failure, which is exactly why LLM reliability has to be defined and measured on its own terms rather than borrowed wholesale from infrastructure reliability practices that were built for a fundamentally different kind of unreliability.

This reframing matters because it directly changes where the actual engineering effort needs to be spent and where a team’s attention needs to go once it stops assuming conventional metrics were ever telling the whole story. Keeping an LLM-based service technically available and responsive is necessary but nowhere near sufficient on its own, and a team that only invests in the infrastructure-reliability half of the problem, retries, load balancing, failover, while leaving the model’s behavioral consistency and its handling of edge cases entirely unaddressed, will end up with a system that looks reliable on every dashboard it built while still disappointing the people using it in ways that dashboard was never designed to catch.

How output consistency becomes its distinct reliability concern

Because an LLM doesn’t guarantee the same output for the same input, consistency has to be treated as a reliability property in its own right, separate from correctness, since a model can be highly consistent while being consistently wrong, or highly variable while still landing on a correct or acceptable answer most of the time, and a reliability practice that only ever measures one of these two dimensions, while entirely ignoring the other, misses problems the other dimension would otherwise have caught and flagged well before they ever reached a user.

Measuring consistency directly and explicitly, running the same or very similar prompts repeatedly over time and carefully checking how much the resulting output varies, whether in substance, in structure, or merely in phrasing, gives a concrete signal that’s independent of whether any single response was correct. A system whose outputs vary wildly in substance across repeated calls on the same input is exhibiting a reliability problem even if every individual output happens to be defensible on its own terms, because a user or a downstream system consuming that output has no way to predict what they’ll get, and that unpredictability itself erodes trust regardless of whether any instance was technically wrong.

Temperature and sampling settings are the single most direct lever available for managing this kind of variability, and the right setting depends heavily on the task at hand: a factual lookup or a structured extraction task benefits from low or zero temperature, where consistency and predictability matter far more than creative variation, while a brainstorming or drafting task may benefit from the variability that a higher temperature introduces, since some diversity across attempts is the actual point in that context rather than a defect to be minimized. Treating temperature as a single global default applied uniformly across every task a system handles, rather than a deliberate, task-choice, is a common and easily avoidable source of reliability problems that trace back to a mismatch between how much consistency a task needs and how much the system’s configuration is providing.

How to measure correctness when there’s rarely a single right answer

Conventional software correctness is usually quite simply binary: a function either returns the right value or it plainly doesn’t, and testing against a fixed set of expected outputs is a well-understood, mechanical process. LLM correctness resists this framing for a large share of tasks, since natural-language output often has many acceptable phrasings, several defensible answers to an open-ended question, or a range of reasonable syntheses of the same underlying material, which means a naive exact-match comparison against a single reference answer will flag a large share of correct outputs as wrong simply because they didn’t happen to match the reference verbatim.

Rubric-based evaluation, carefully defining the properties a correct or acceptable answer needs to exhibit rather than a single canonical answer it must match exactly, handles this considerably better, letting an evaluation process check whether a response covers the necessary facts, avoids known errors, and stays within acceptable bounds of tone and structure, without requiring it to match one particular phrasing exactly. Model-based evaluation, using a separate, dedicated model call specifically tasked with judging whether a response satisfies a rubric or is faithful to a reference, extends this further and scales considerably better than manual review, though it introduces its reliability question one level up, since the evaluating model is itself subject to the same non-determinism and the same potential for error as the model being evaluated, which is exactly why evaluation methodology deserves its validation against a smaller set of human-judged examples before it’s trusted at scale.

Task-correctness criteria built around what matters for an application rather than a generic notion of “good” output, matter more here than in most conventional software testing, since a factual question-answering task and a creative-writing task and a code-generation task each have different notions of what correctness even means, and a single, generic evaluation approach applied uniformly across all three will inevitably be poorly calibrated for at least one of them, if not for all three simultaneously.

How hallucination undermines reliability in a way that’s specific to language models

Hallucination, a model generating content that’s fluent, entirely plausible, and confidently stated but not grounded in fact or in the material it was originally given, is a failure mode with no close analogue anywhere in conventional software, since a conventional program doesn’t simply invent a plausible-sounding but false result the way a language model can, and this makes hallucination one of the most consequential and most distinctive reliability concerns specific to LLM-based systems.

Hallucination is particularly damaging to reliability precisely because it doesn’t announce itself, a hallucinated answer reads exactly like a well-supported one, with the same fluent, confident tone, which means a user has no inherent way to distinguish a reliable answer from a fabricated one purely by reading it, and this is exactly why grounding, retrieval-augmented generation, explicit citation requirements, and honest uncertainty signaling all matter as reliability practices specifically, not merely as quality-of-life improvements, since they address a failure mode that’s uniquely capable of looking exactly as trustworthy as a reliable response.

Reducing hallucination reliably requires attacking it from several directions simultaneously rather than expecting any single technique to solve it on its own: grounding responses in retrieved, verifiable material rather than the model’s unconstrained generation reduces the opportunity for fabrication in the first place, explicit instructions to acknowledge uncertainty rather than filling gaps with plausible-sounding invention reduce the model’s tendency to paper over what it doesn’t know, and dedicated evaluation specifically targeting faithfulness, checking whether a response’s claims are supported by what it was given, catches what slips through the first two layers before it ever reaches a user who has no independent way to catch it themselves.

How reliability differs across single-turn responses and longer, multi-step interactions

A single, isolated model call answering a single self-contained question has a comparatively narrow surface for reliability problems to enter, mainly the correctness and consistency concerns already discussed, but a longer interaction, a multi-turn conversation or an agentic sequence involving several dependent steps, introduces reliability risks that compound across the length of the interaction in ways a single call never has the opportunity to.

Context accumulation across a longer interaction creates its reliability risk, since an early error or an early ambiguity that goes uncorrected can propagate forward and shape every subsequent turn, meaning a single early reliability lapse doesn’t just affect one response, it can quietly degrade the reliability of everything that follows it within the same interaction, which is exactly why catching and correcting an error early in a longer interaction matters disproportionately more than catching the equivalent error in an isolated, single-turn exchange. Multi-step agentic sequences compound this further still since each additional step carries its independent chance of introducing an error, and the overall reliability of a sequence isn’t simply the reliability of its best or even its average step, it’s closer to the product of each individual step’s reliability, which means a sequence of several individually quite reliable steps can still have a meaningfully higher overall chance of ending up wrong somewhere along the way than any single step’s reliability would suggest in isolation.

This compounding effect is exactly, precisely why reliability practices built only around evaluating a system’s final output alone, without any visibility into the intermediate steps that produced it, systematically underestimate how much reliability risk exists in a longer interaction, and it’s why the stage-evaluation discussed elsewhere in AI system design, checking retrieval quality, checking individual reasoning steps, checking tool-call correctness, each independently rather than only checking the polished final answer, matters disproportionately more as interactions grow longer and involve more dependent steps.

How reliability practices need to account for model updates and drift over time

An LLM-based system’s actual reliability isn’t a fixed property that gets established once during initial development and then simply holds steady indefinitely afterward, because the underlying model itself can change, through a provider’s updates, independently of anything the team building on top of it modified, which means a system’s actual, measured reliability can shift even when the surrounding application code, prompts, and configuration remain completely untouched.

Version pinning, in the cases where a provider makes it available, gives a team explicit control over exactly when a model change takes effect, letting that change go through the same evaluation process any other significant system change would go through rather than arriving unannounced and untested, and this control matters specifically because a model update that improves reliability on most tasks can still introduce a regression on some particular task type a team’s evaluation suite was never built to specifically check. Continuous evaluation running a system’s reliability checks on an ongoing, sustained basis rather than merely once during the initial launch period, is what catches drift that happens for reasons entirely outside a team’s control, whether from an unpinned model update, a shift in the distribution of user queries away from whatever the original evaluation set covered, or gradual changes in upstream data that quietly alter what the model receives even when nothing about the model itself has changed.

Treating a reliability evaluation performed months ago as still informative but no longer fully sufficient evidence that a system remains reliable today is the mindset that protects a team against this kind of drift, and it’s a different mindset from the conventional software testing practice of trusting a passing test suite indefinitely as long as the underlying code hasn’t changed, precisely because an LLM-based system’s actual behavior can shift for reasons that have nothing to do with the code changing at all.

How uncertainty and confidence signals become a core reliability mechanism

A model that always responds with exactly the same confident tone regardless of how well-supported its actual answer happens to be offers no way for a downstream system or a user to ever distinguish a reliable response from a considerably shakier one, which means honest uncertainty signaling isn’t a nice-to-have refinement, it’s a core mechanism reliability depends on, since a system incapable of ever expressing doubt is, in an important sense, systematically less reliable than one that can, even if the two systems’ raw correctness rates happened to be identical.

Building reliable confidence signals into a system requires considerably more than simply instructing a model to hedge whenever it happens to feel uncertain, since a model prompted this way can still fail to recognize the situations that warrant hedging, which is why the more reliable approach combines explicit uncertainty instructions with structural signals available elsewhere in the pipeline: how strong the underlying retrieved evidence was in a retrieval-augmented system, how many ambiguous or uncertain decision points a task passed through in an agentic sequence, or how far a query sits from the kind of input a system’s evaluation suite has validated it against. These structural signals, generated independently of the model’s self-reported confidence, give a system a way to cross-check whether a confidently phrased response is backed by confidently strong underlying support, rather than relying entirely on the model’s potentially miscalibrated sense of its certainty.

Acting on these confidence signals once they exist is precisely what turns them into a reliability improvement rather than an unused, merely decorative diagnostic, whether through the graduated response and graceful degradation patterns relevant to AI system design more broadly, routing low-confidence responses toward additional verification, a more conservative answer, or an honest acknowledgment of insufficient information, rather than letting every response, regardless of its actual underlying confidence, reach a user in exactly the same uniformly assured presentation.

How to build a reliability testing practice that catches problems

A reliability testing practice built only around a small, static set of example queries checked exactly once before launch catches only the narrow failure modes that happened to be anticipated well in advance, which is a considerably narrower net than what production traffic will eventually test a system against, and building a testing practice that catches the reliability problems that matter requires deliberately going beyond this narrow, static starting point.

Adversarial and edge-case testing, deliberately, carefully constructing inputs designed to probe the failure modes already discussed throughout this article, queries likely to trigger hallucination, inputs at the boundary of what a system’s underlying data covers, prompts phrased ambiguously enough to invite inconsistent interpretation, exercises the reliability mechanisms a system depends on far more directly than testing only against typical, well-behaved queries ever will, since typical queries are exactly the ones a reasonably well-built system is least likely to struggle with in the first place. Longitudinal testing, checking not merely whether a single response is correct but whether repeated calls on the same or similar inputs stay consistent over time and across separate sessions, catches the consistency-related reliability problems that a single-shot evaluation, checking only one response per test case, structurally can’t detect at all, since consistency by definition requires observing multiple responses to the same or similar input rather than judging any single response in isolation.

Production monitoring that closes the loop back to this testing practice, feeding observed failure patterns and observed edge cases back into the test suite on an ongoing, sustained basis rather than treating the test suite as a fixed artifact finalized once during initial development, is what keeps a reliability practice representative of the failure modes a system encounters, rather than gradually drifting out of touch with production behavior as that behavior itself continues to evolve in ways the original, static test suite was never built to anticipate.

How reliability engineering connects to the broader discipline of AI native system design

Everything discussed at length throughout this article, non-deterministic output, hallucination, compounding multi-step risk, model drift, honest uncertainty signaling, all reflects a single, shared underlying theme: an LLM-based component behaves in ways a conventional software component simply doesn’t, and reliability engineering for it has to be built around that difference explicitly rather than adapted awkwardly from practices designed for components that behave in fundamentally more predictable ways.

This is exactly, precisely the same underlying insight that already drives AI native system design more broadly, treating uncertainty, context management, and continuous evaluation as first-class architectural concerns rather than edge cases handled reactively once they surface in production, and LLM reliability specifically is best understood as one concrete, practical expression of that broader design philosophy rather than as a separate discipline unto itself. A team that has deeply internalized AI native design principles at the architectural level will consistently find that most of the reliability practices discussed here, stage-evaluation, honest uncertainty communication, continuous rather than one-time testing, aren’t bolted-on additions to an otherwise conventional system, they’re simply the natural consequence of having designed the surrounding architecture correctly around the model’s behavior from the very start.

How reliability requirements differ across different kinds of applications

Not every single LLM-based application needs the same degree, or even the same kind, of reliability, and treating reliability as a single universal bar every system must clear equally misses an important design question: what goes wrong, and how badly, if this particular system produces an unreliable response, and the answer to that question varies enormously depending on what the system is being used for and who’s relying on its output.

A creative-writing assistant or an open-ended brainstorming tool can tolerate considerable output variability, and in fact often benefits from it, since the entire value proposition of a tool like that depends on diversity across attempts rather than convergent, repeatable output, which means the consistency concerns discussed earlier apply with much less force here than they do elsewhere. A system generating factual summaries, answering customer support questions, or producing content a reader will treat as authoritative sits at the opposite, considerably more demanding end of this spectrum, where hallucination and inconsistency carry reputational and sometimes financial cost, and where the reliability investment discussed throughout this article isn’t optional polish, it’s a prerequisite for the system being fit for its actual purpose at all.

Systems that take real-world, consequential action, an agentic system able to send messages, modify records, or execute transactions directly on a user’s behalf, sit in a still more demanding category, since an unreliable response here doesn’t merely mislead a reader, it can produce an incorrect, sometimes irreversible action in the world, which is exactly, precisely why the confidence-aware routing and human-escalation patterns discussed elsewhere deserve to be applied with seriousness specifically in this particular category, rather than treated as some optional refinement appropriate mainly for the very highest-stakes systems a team happens to be building at the time. Carefully calibrating how much reliability investment a system needs, rather than either under-investing in a system whose failures carry consequences or over-investing in one whose variability is part of its core value, is itself a meaningful part of designing LLM reliability well, and it’s a calibration that has to be made deliberately for each application in turn, rather than simply assumed to be the same, uniform value across every single system a team happens to operate.

How organizational practices need to support reliability as an ongoing commitment

LLM reliability is not a property a team achieves just once and then simply maintains passively thereafter, with no further attention required, because everything discussed throughout this article, model drift, evolving traffic patterns, the compounding risk of longer interactions, requires ongoing attention rather than a one-time engineering investment that’s considered complete once a system first launches successfully.

Clear, explicit ownership over reliability evaluation and monitoring, a person or a team directly responsible for maintaining and actively expanding the test suite, tracking production reliability metrics, and responding when those metrics start drifting in a concerning direction, matters considerably more here than it does for conventional software, precisely because the causes of an LLM-based system’s reliability degradation are so often external to any change the team itself made, an unpinned model update, a shift in user behavior, which means nobody will notice the degradation at all unless someone is watching for it on an ongoing basis rather than only during the initial launch period. Incident response processes deserve a similar, equally deliberate update in the same spirit, since a team’s existing playbook, built around detecting and diagnosing a conventional, cleanly bounded failure, may have no procedure at all for the investigative work an LLM reliability incident requires, tracing a pattern of subtly wrong responses back to whichever stage, retrieval, a particular prompt component, an unpinned model update introduced the regression.

Product and engineering decisions about exactly what level of reliability is acceptable, how much output variability is tolerable for a feature, how aggressively the system should hedge when uncertain, how much extra cost is truly worth spending on additional verification for a higher-stakes task, deserve to be made deliberately and revisited periodically as a system, its traffic, and the underlying models it depends on all continue to evolve, rather than left as implicit defaults nobody explicitly chose and nobody is responsible for reconsidering as circumstances change around them over time.

Common mistakes teams make around LLM reliability

A first mistake, and the foundational one nearly every other mistake on this list traces back to in some form, is measuring LLM reliability using conventional infrastructure metrics alone, uptime, error rate, latency, while leaving the model’s behavioral consistency and correctness entirely unmeasured, mistaking mere technical availability for the actual, substantive reliability that matters to the people relying on the system’s output.

A second mistake is treating output consistency and output correctness as though they were the exact same underlying property, missing the cases where a system is highly consistent while consistently, reliably wrong, or variable while still landing on an acceptable answer most of the time regardless of that variability.

A third mistake is applying a single, global temperature or sampling configuration uniformly across every distinct task a system handles, mismatching how much consistency a task needs against how much the system’s configuration is providing it in practice.

A fourth mistake is evaluating correctness through a naive exact-match comparison against one single reference answer, flagging a large share of correct, merely differently phrased responses as wrong simply because they happened not to match the reference text verbatim, word for word.

A fifth mistake is trusting model-based evaluation entirely, without ever independently validating its judgments against a smaller set of carefully human-reviewed examples, overlooking the basic fact that the evaluating model is itself subject to the exact same non-determinism and the exact same potential for error as whatever model is being evaluated by it.

A sixth mistake is treating hallucination as merely an occasional quality glitch rather than the distinctive structural reliability risk it is, underinvesting in grounding, dedicated faithfulness evaluation, and honest uncertainty signaling specifically because hallucinated output reads exactly, indistinguishably as confidently as reliable output does.

A seventh mistake is evaluating only a system’s final output in a longer multi-step interaction, missing the compounding risk that accumulates across dependent steps and the considerably disproportionate value of catching an early error before it ever has the chance to propagate forward.

An eighth mistake is treating a reliability evaluation performed once, months ago, as still sufficient evidence that a system remains reliable today, ignoring the ways model updates, shifting traffic patterns, and gradual upstream data drift can all quietly degrade a system’s actual reliability with no corresponding change to a team’s code whatsoever.

A ninth mistake is simply instructing a model to hedge when uncertain without ever backing that instruction with any independent structural confidence signal of its own, leaving the system with no way to cross-check whether a confidently phrased response is backed by confidently strong underlying support in the first place.

A tenth mistake is going to the trouble of generating confidence signals but then never acting on them at all, computing a meaningful uncertainty measure and then routing every single response through exactly the same presentation regardless of how much underlying confidence that measure reflects.

An eleventh mistake is building a reliability test suite exactly once, from a small, static set of anticipated example queries chosen in advance, and then never expanding it with observed failure patterns and edge cases as those cases surface in production traffic over time.

A twelfth mistake is testing only single-shot correctness with no longitudinal testing of consistency across repeated calls, structurally missing the reliability problems that only ever become visible once the same or a similar input is observed more than a single time.

A thirteenth mistake is applying exactly the same reliability bar uniformly across every different application a team happens to build, over-investing in a creative or exploratory tool whose variability is part of its value, or badly under-investing in a consequential, action-taking system whose failures carry real, and sometimes irreversible, cost.

A fourteenth mistake is treating reliability as a one-time engineering investment considered complete at launch, with no clear ongoing ownership over evaluation and monitoring afterward, leaving external drift, an unpinned model update, a shift in traffic, free to degrade actual reliability with nobody watching for it as it happens.

A fifteenth and truly final mistake is relying entirely on an incident response process originally built for conventional, cleanly bounded failures to handle an LLM reliability incident, with no dedicated procedure for tracing a pattern of subtly wrong responses back to whichever stage originally introduced the regression in the first place.

What connects all fifteen of these mistakes is applying reliability thinking built specifically for deterministic, conventional software to a component that simply, fundamentally doesn’t behave that way, and expecting the resulting practice to somehow hold up regardless of that basic, unavoidable mismatch between assumption and reality. Teams that instead build reliability practices specifically around an LLM’s actual behavior, consistency measured on its own terms, correctness evaluated against rubrics rather than rigid exact matches, hallucination treated as a structural risk rather than an occasional glitch, confidence signals that are both generated and acted upon, tend to build systems that stay trustworthy under sustained production use, while teams that skip this deliberate redesign tend to discover the gap only once a user encounters a confidently wrong answer that every one of their conventional reliability dashboards said should never have happened.

The deeper lesson underlying all of this is that reliability, for a system built around a language model, is never a fixed property you achieve once and then simply preserve, it’s an ongoing relationship between the model’s non-deterministic behavior and the surrounding system’s willingness to account for that non-determinism rather than paper over it. A system that measures consistency honestly, evaluates correctness against rubrics rather than brittle exact matches, treats hallucination as the structural risk it is, and keeps its testing and monitoring practices actively current rather than frozen at launch will, over time, earn a level of trust that no amount of conventional infrastructure hardening alone could ever produce on its own, precisely because that trust is earned through consistently honest behavior under varied conditions rather than through the mere absence of visible outages.

Teams that internalize this early, building reliability directly into the architecture itself rather than bolting monitoring onto a system that was never designed with the model’s non-deterministic behavior in mind from the very start, tend to end up with systems that hold up honestly under sustained production use, failing informatively and gracefully on the cases that warrant caution rather than either confidently misleading their users or refusing to help on cases the system could have handled perfectly well.

Reliability, understood this way, becomes less a checkbox to tick off before launch and more a continuous practice woven into how a team builds, evaluates, and operates its AI native systems day to day, a practice that pays off precisely in the moments a system is under unplanned strain, the exact moments a conventional reliability dashboard, built for a fundamentally different kind of software, was never equipped to see coming in the first place, and the exact moments the people relying on the system are least prepared to absorb a confidently wrong answer without lasting consequence.