What is agent reliability?

Summary slides

Why agent reliability is a different problem

A single-shot AI system — one that takes an input and produces one output in one pass — has exactly one place its reliability can fail: the generation step itself. An agent, by contrast, might take five, ten, or twenty separate actions to complete a single task, and at every one of those steps, it has to make a correct decision about what to do next, correctly execute whatever action it decided on, and correctly interpret the result before moving to the next step. If each individual step has, say, a ninety-five percent chance of going right — which sounds like a strong number for any individual decision — a ten-step task where every step needs to succeed has roughly a sixty percent chance of completing correctly end to end, purely from that per-step error compounding, without anything in particular going unusually wrong at any single step along the way.

This compounding effect is the central fact that makes agent reliability its discipline rather than simply an application of general AI reliability principles to a longer task. It means that a per-step failure rate that would be completely unremarkable in a single-shot system becomes a serious end-to-end reliability problem once that same step is repeated many times across a long agentic task. It also means that improving agent reliability is rarely about finding one big fix — it’s almost always about identifying and reducing the failure rate at each individual step in the chain, because the total reliability of the system is the product of all of them together, and a weak link anywhere in that chain drags down the whole task’s success rate regardless of how strong every other step is — which is exactly why the sections that follow treat each failure point as its distinct problem rather than lumping them together under one general notion of “the agent sometimes gets things wrong.”

Action selection: choosing the wrong next step

Given that every step in the chain matters, the first place to look at where things go wrong is the very first decision an agent makes at each step: what to do next. Action selection failures happen when an agent, reasoning about the current state of a task, picks a tool or action that isn’t the right one — reaching for a search tool when a calculation was needed, calling the wrong specialist agent in a multi-agent system, or repeating an action it already tried and that already failed, because its reasoning didn’t correctly register that the previous attempt hadn’t worked.

This failure mode gets meaningfully worse as the number of available actions grows, which connects directly back to how an agent’s tools are designed in the first place: an agent choosing between three clearly distinguished, narrowly scoped tools has an easier and more reliable decision to make than one choosing between fifteen tools with overlapping or ambiguously described purposes. It also gets worse when the current state of a task is ambiguous or under-communicated to the agent — if the reasoning step doesn’t have a clear, accurate summary of what’s already been tried and what the results were, it’s making its next decision partly blind, which is precisely the condition under which an agent repeats a failed action or picks an action that’s already been ruled out by something learned earlier in the task.

Tool execution: the action was right, but it didn’t work

Even when an agent correctly decides what to do next, the action itself still has to execute successfully, and this is a distinct failure point worth separating cleanly from action selection, because the fixes for each are completely different. Tool execution failures include a tool call that’s malformed — the agent generated a request with the wrong parameters, a missing required field, or a format the underlying tool can’t parse — a tool that’s temporarily unavailable or rate-limited, or a tool that executes successfully but returns a result in a form the agent’s next reasoning step doesn’t correctly interpret.

Malformed tool calls are often a design problem more than a model-capability problem: a tool with a clean, simple, well-documented interface and a clear description of exactly what parameters it expects gets called correctly far more often than a tool with a vague or overly flexible interface that leaves more room for the agent to generate something subtly wrong. Transient unavailability — a rate limit, a brief outage, a timeout — is a different kind of problem entirely, one that has nothing to do with whether the agent’s decision was sound and everything to do with whether the system has retry logic in place that treats a transient failure as retryable rather than as a dead end that derails the whole task. And misinterpreted results are often the quietest of these three failure types, because nothing visibly breaks — the tool call succeeds, returns data, and the agent simply reasons incorrectly about what that data means, carrying a wrong conclusion forward into every subsequent step without any error ever being raised along the way, which makes it one of the hardest of these failure types to catch through ordinary error monitoring, since nothing in the system’s logs looks unusual even though the reasoning built on top of that misread result is quietly wrong from that point forward.

Planning and reasoning: getting the strategy wrong, not just one step

Action selection and tool execution both operate at the level of a single step, but agents also make a higher-level kind of decision that spans the whole task: the overall plan or strategy for how to approach it, and this is a failure point of its own, distinct from any individual step going wrong. A plan can be flawed from the start — an agent might commit early to an approach that seems reasonable but turns out to be a dead end several steps in, and depending on how the agent’s reasoning loop is structured, it may not recognize that dead end and reconsider its approach until it’s wasted considerable time and several actions pursuing it.

This shows up in a recognizable pattern sometimes called loop failure: an agent gets stuck repeating a similar sequence of actions without making progress, often because its reasoning keeps arriving at a similar, plausible-but-wrong conclusion about what to try next, without anything in its process forcing a different approach once the first few attempts haven’t worked. It also shows up as premature convergence — an agent that decides it has enough information to answer before it does, skipping steps that a more careful approach would have taken — which is a subtler failure than a loop, because the agent completes the task and produces an answer, but that answer is built on an incomplete investigation the agent itself never registered as incomplete. Both failure types point to the same underlying design need: an agent’s reasoning loop needs an explicit mechanism for recognizing when its current approach isn’t working and needs to change, rather than relying entirely on the model’s judgment to notice that on its own, since that judgment is exactly the thing that’s already failing when a loop or premature convergence happens in the first place.

Memory and state: losing track of what’s already happened

As a task runs longer, an agent’s reliability increasingly depends on something that has nothing to do with any single decision: whether it’s correctly keeping track of what’s already happened. State and memory failures happen when an agent loses, forgets, or misremembers something established earlier in the task — a constraint the user specified at the start, an intermediate result from three steps ago, a fact it already verified and shouldn’t need to re-check. These failures tend to get worse as task length grows, for the same reason discussed elsewhere in this category of system: longer contexts dilute a model’s attention across more information, and information that isn’t actively surfaced in a compact, relevant form at each reasoning step is more likely to get effectively lost even if it’s technically still present somewhere in the accumulated history.

The practical fix here is less about the model itself and more about how an agent’s memory is architected: maintaining a compact, actively updated summary of task progress — what’s been tried, what’s been learned, what constraints are still in force — rather than relying on the model to correctly re-derive that summary from a growing, unstructured history at every single step. An agent that has to re-read and re-interpret its entire raw history at every reasoning step is doing more work, and more error-prone work, than one that’s handed a maintained, current summary of exactly the state it needs to make its next decision well. This is a case where a small amount of deliberate engineering — actively managing what an agent carries forward, rather than just letting history accumulate — has an outsized effect on end-to-end reliability, precisely because state failures compound across every subsequent step once they happen, the same way any other per-step failure does.

Knowing when to stop: finishing too early, too late, or never

Every one of the failure types covered so far assumes the agent is still actively working through a task, but a distinct and equally consequential failure point is whether it correctly recognizes when to stop — and this turns out to be one of the harder problems in agent design precisely because “done” doesn’t have an obvious, checkable definition the way it might for a simpler system. An agent that stops too early produces an incomplete answer with the same confident tone as a complete one, which is arguably worse than an agent that runs long, because there’s often no visible signal to a user that the task wasn’t finished. An agent that doesn’t recognize a natural stopping point at all keeps taking actions well past the point of diminishing returns, burning time and cost without making additional progress, and in the worst case never terminates at all without a hard limit forcing it to.

Reliable stopping behavior generally needs to be more than an implicit judgment left entirely to the model’s sense of “am I done” — it benefits from an explicit check, sometimes a separate verification step that specifically evaluates whether the accumulated work satisfies the original task’s requirements before the agent commits to a final answer, rather than trusting the same reasoning process that’s been making per-step decisions all along to also correctly self-assess its completeness. Combined with a hard step or time ceiling as a backstop — not as the primary stopping mechanism, but as a safety net for the cases where the primary mechanism fails to trigger — this turns “knowing when to stop” from an implicit, unreliable judgment call into an explicit, checkable part of the system’s design.

Measuring agent reliability in practice

Understanding all of these failure points matters practically only if there’s a way to measure how often each one occurs and whether changes to the system are making things better or worse, and agent reliability measurement needs to go beyond a single end-to-end success rate to be useful for diagnosis. Task success rate — did the agent complete the task correctly, evaluated against a representative set of realistic tasks — is the most important top-line number, but on its own it doesn’t say anything about where a failure originated, which matters enormously for fixing the problem rather than just knowing it exists.

Step-level metrics fill that gap: action selection accuracy, tracked separately from tool execution success, tracked separately from whether the agent’s final self-assessment of completeness matched an independent evaluation of whether the task was done well. Efficiency metrics matter too, and not only for cost reasons — the number of steps an agent takes to complete a task, and how that compares to a reasonable minimum, is itself a reliability signal, because an agent that reliably completes tasks but takes twice as many steps as necessary is very often burning some of those extra steps on the exact loop and recovery patterns described earlier, even in runs that technically still end in a correct result. And tracking these metrics across different task types and complexity levels separately, rather than as one blended average, matters because an agent’s reliability profile is rarely uniform — a system that handles simple, short tasks reliably can still have a much lower success rate on longer, more complex tasks, and averaging the two together hides exactly the information a team needs to know where reliability work should be focused.

Trace-level review deserves its mention here, distinct from any aggregate metric: for any individual failed task, being able to look at the full sequence of actions, tool calls, and reasoning steps the agent took is what turns “this task failed” into “this task failed because the agent misread a tool result at step four and built the rest of its plan on that misreading” — a level of diagnostic detail no aggregate success-rate number can provide on its own, and one that turns a vague sense that a system “needs improvement” into an actionable engineering task. Building this kind of structured, reviewable tracing into an agent from the start is considerably cheaper than trying to retrofit it onto a system that’s already live and already producing failures nobody can fully explain, which is the same lesson that applies to observability in AI systems more broadly, just doubly important here given how many more individual decision points an agent’s trace contains compared to a single-shot system’s.

Designing for reliability rather than hoping for it

Everything covered so far describes where and why agent reliability breaks down; the corresponding design response is to build explicit structure around each of those failure points rather than relying on a capable model to simply avoid them through good judgment alone, since that judgment is exactly what’s shown to be unreliable at the per-step level this whole discussion has been built around. Narrow, well-documented tools reduce action selection errors directly, for the same reasons discussed earlier. Retry logic with sensible backoff handles transient tool failures without derailing an entire task over what was really just a momentary, recoverable glitch. An explicit, actively maintained state summary reduces memory failures. A separate completeness-verification step, distinct from the agent’s in-the-moment reasoning, catches premature stopping before it reaches a user.

A pattern worth calling out specifically because it addresses several of these failure types at once is adding a lightweight self-check after key steps — not a full separate agent, but a deliberate pause where the system verifies that the last action’s result is consistent with what was expected before committing to build the next step on top of it. This catches misinterpreted tool results and reasoning drift early, before an error has had the chance to compound across several more steps the way an unnoticed early mistake otherwise would. None of these are exotic techniques — they’re the same kind of deliberate, explicit engineering that makes any complex system reliable — but applying them specifically to the per-step failure points that make agents different from single-shot systems is what turns general reliability engineering into agent reliability engineering in particular.

Why a more capable model doesn’t fully solve this on its own

Given how much of this compounding-error problem comes down to per-step decision quality, it’s a reasonable question whether the whole thing is really just a matter of waiting for more capable models — if each step’s error rate improves as models get better, doesn’t the end-to-end reliability problem shrink on its own without needing any of the explicit design work described above? The honest answer is: it helps, meaningfully, but it doesn’t fully substitute for deliberate reliability engineering, for a couple of reasons worth being precise about.

The first is that the compounding math means even large improvements in per-step accuracy produce comparatively modest improvements in end-to-end reliability once a task involves enough steps — moving a per-step success rate from ninety-five to ninety-eight percent sounds like a strong, meaningful improvement, and it is, but across a fifteen-step task it moves the end-to-end success rate from roughly forty-six percent to about seventy-four percent, a welcome gain, yet still far short of the reliability bar most production systems need to hit for tasks of any length. Model improvements alone, without any of the explicit structural changes described throughout this page, tend to raise that per-step ceiling gradually — they don’t eliminate the compounding effect itself, because that effect is a property of how many independent decisions a task requires, not a property of any single model’s capability — and no amount of raw capability improvement changes how many separate decisions a complex task requires along the way.

The second reason is that several of the failure modes covered here aren’t really capability problems in the sense that a stronger model directly fixes them. A more capable model can still be handed a vague, overlapping set of tool descriptions and make an action-selection mistake, because the ambiguity lives in the tool design, not in the model’s reasoning ability. A more capable model can still lose track of an early constraint in a long task if the system hands it an unstructured, ever-growing history rather than an actively maintained summary, because the failure is about how state is managed around the model, not about how well the model itself reasons over whatever it’s given. A more capable model can still fail to recognize a coordination gap between two agents in a multi-agent system, because that gap lives in the handoff design between agents, not inside either individual agent’s reasoning. This is why the practical, durable answer to agent reliability has consistently been a combination of model improvement and deliberate system design working together, rather than treating either one as sufficient on its own — model improvement raises the ceiling on what’s achievable, and system design is what gets a production system close to that ceiling rather than leaving a meaningful gap between what the underlying model is capable of and what the system built around it reliably delivers.

Recovery: what happens after something goes wrong

Designing to reduce how often each failure type occurs only gets a system so far, because at usage volume, across enough tasks, something will eventually go wrong regardless of how well the system is designed — which makes recovery strategy just as important as prevention, and worth designing with the same deliberateness. A tiered recovery approach tends to work well in practice: for a transient, likely-recoverable failure — a tool timeout, a momentary rate limit — a straightforward retry, sometimes with a slightly adjusted approach, is often sufficient and doesn’t need to involve the user or a human operator at all.

For a failure that persists after a retry — a tool that keeps failing, a plan that’s reached a dead end — the more productive response is usually to have the agent explicitly reconsider its approach, informed by what specifically didn’t work, rather than mechanically retrying the identical failed action again and expecting a different result. And for a failure the agent can’t recover from on its own — missing information no available tool can supply, a request that’s ambiguous in a way only a human can resolve — the most reliable systems are designed to recognize that condition and escalate honestly, reporting what was attempted and why it didn’t succeed, rather than either failing silently or generating a plausible-sounding answer that quietly papers over the fact that the task was never completed at all. This tiered structure — retry, reconsider, escalate — mirrors how a careful human works through the same kind of trouble, and building it in explicitly, rather than leaving recovery behavior to whatever an agent happens to do by default, is one of the most reliable ways to keep individual step failures from turning into full task failures. Without this structure in place, a system’s default behavior when something unexpected happens is whatever falls out of the model’s reasoning in that moment, which can range from a reasonable improvised recovery to a confused, unproductive spiral, and leaving that outcome to chance rather than to deliberate design is exactly the gap a tiered recovery strategy exists to close.

Multi-agent systems: a new coordination failure surface

Everything discussed so far applies within a single agent’s reasoning loop, but many systems split work across multiple agents — a supervisor delegating to specialists, agents handing off work to each other in sequence — and this introduces a category of reliability failure that doesn’t exist at all in a single-agent system: coordination failure between agents, layered on top of every per-step failure type already covered within each individual agent’s loop. A handoff between two agents is itself a point where information can be lost or misrepresented — if a supervisor agent summarizes a task for a specialist agent, and that summary drops a constraint the user originally specified, the specialist is now working from an incomplete brief through no fault of its reasoning, and nothing in its loop would necessarily catch that the brief it received was already deficient.

Miscommunication of this kind compounds in a way that’s harder to diagnose than a single agent’s internal error, because the failure doesn’t live inside any one agent’s reasoning trace — it lives in the gap between two agents’ traces, in exactly the information that didn’t successfully cross the handoff. This is why well-designed multi-agent systems tend to define handoffs as structured, explicit interfaces rather than free-form natural-language summaries alone — a specialist agent that receives a structured task specification, with required fields the supervisor has to fill in explicitly, is considerably less likely to silently lose a constraint than one that receives whatever unstructured summary the supervisor’s reasoning happened to produce in the moment. A second coordination failure mode is inconsistency between agents that are each individually reasoning correctly but arrive at conclusions that don’t fit together — one specialist assumes a piece of context that a second specialist’s output contradicts, and without an explicit reconciliation step, that contradiction can persist all the way into a final combined output that reads as coherent while containing an internal inconsistency neither individual agent’s reasoning process was positioned to notice. Multi-agent reliability, in short, needs everything single-agent reliability needs, applied within each agent’s loop, plus an entirely additional layer of attention paid specifically to the handoffs and interactions between agents — a layer that has no analogue at all in a single-agent system and is easy to underweight precisely because it doesn’t show up clearly in any single agent’s internal reasoning trace. Teams that skip this extra layer of attention tend to discover it the hard way: every individual agent’s trace looks reasonable in isolation, and yet the combined, final output is subtly wrong, which is a confusing failure mode to debug until a team specifically starts looking at the handoffs themselves as a distinct point of potential failure, rather than only ever inspecting each agent’s reasoning on its own.

A worked example: a research agent gathering information across sources

These principles are easier to see clearly against a concrete case than left purely abstract, so consider an agent tasked with researching a topic across several internal knowledge sources and producing a synthesized summary — a task that’s long enough, and involves enough separate steps, to make every failure point discussed here relevant. Action selection reliability here means the agent reliably chooses which source to search based on what kind of information is still missing, rather than repeatedly searching the same source hoping for a different result. This is supported directly by keeping each source behind its narrowly described tool, rather than one generic “search everything” tool that leaves the choice of source ambiguous.

Tool execution reliability means each individual search call is retried automatically on a transient failure, with the agent’s reasoning explicitly informed when a source is unavailable so it can route around that gap rather than silently proceeding as if the missing source had simply returned nothing relevant. Planning reliability shows up in how the agent tracks what’s already been covered — an actively maintained summary of what’s been found on each subtopic — so that it recognizes when it has sufficient coverage across all the subtopics the research question requires, rather than either stopping after covering only the first subtopic it happened to investigate or continuing to search well past the point where additional searches were adding anything new. A completeness check runs before the agent commits to its final summary, explicitly comparing what was found against what the original question asked for, catching the case where a subtopic was accidentally never investigated at all. And if a necessary source turns out to be unavailable for the whole research session, the agent reports that limitation explicitly in its final summary — “the following was not covered because source X was unavailable” — rather than presenting a summary that looks complete while quietly omitting exactly the part it couldn’t investigate.

Common mistakes in how teams approach agent reliability

A consistent set of mistakes shows up across teams building agentic systems, and they’re worth naming directly because each one traces back to treating agent reliability as if it were the same problem as single-shot system reliability, when the compounding, multi-step nature of agents makes it a meaningfully different one. The most common is measuring only end-to-end task success and never breaking failures down by which step they originated at, which means a team can see that reliability is a problem without having any path to fixing it, since the fix for an action-selection problem, a tool-execution problem, and a stopping-condition problem are all completely different from each other.

A second common mistake is under-investing in tool design specifically, treating tools as simple wrappers around existing functionality rather than as interfaces that need to be well-designed for reliable agent use — vague descriptions, overlapping purposes, and permissive parameter formats that accept almost anything all quietly increase the rate of action-selection and tool-execution failures in ways that are easy to trace back to the tools themselves once you know to look there, but easy to miss otherwise. A third is building no explicit stopping-condition verification at all, relying entirely on the agent’s in-the-moment judgment about whether it’s done, which is precisely the judgment shown throughout this page to be unreliable exactly when a task has gone even slightly off track. A fourth is designing recovery behavior reactively, only after a failure embarrasses a team in production, rather than building the retry-reconsider-escalate structure in from the start as a first-class part of the system. And a fifth, tying back to where this page began, is simply underestimating how much per-step error compounds across a long task — treating a ninety-five percent per-step success rate as “basically reliable” without doing the arithmetic that shows exactly how much that number degrades once it’s multiplied across every step a realistic task requires, which is the single most consistent gap between how reliable a demo looks and how reliable the same agent turns out to be once it’s handling tasks at length and volume.

That gap between demo and production is really the thread running through everything covered here. A demo task is usually short, runs a handful of steps, and gets tried a small number of times by people who already know roughly what to expect — conditions under which per-step error rates barely have room to compound into anything visible. A production agent runs the same underlying reasoning process across far more steps, far more varied inputs, and far higher volume, which is exactly the setting where a per-step error rate that looked negligible in testing turns into a measurable end-to-end reliability problem. Closing that gap isn’t about finding one clever fix — it’s the accumulated effect of treating action selection, tool execution, planning, memory, and stopping conditions as separate, individually measurable and individually improvable failure points, building explicit recovery behavior rather than hoping failures don’t happen, and — for any system spanning more than one agent — paying just as much deliberate attention to the handoffs between agents as to the reasoning inside any single one of them. None of this is exotic engineering, but skipping it is the most reliable way to end up with a system that performed beautifully in every demo and then disappointed everyone the moment it met sustained, real-world use.