How do you design agentic systems?

Summary slides

What makes a system “agentic”

Before any of those design decisions make sense, it’s worth being precise about what separates an agentic system from a regular AI-powered pipeline, because that distinction is what drives almost every choice that follows. A traditional pipeline — even one built around a language model — follows a fixed sequence of steps decided in advance by the developer: retrieve, then summarize, then format, always in that order, regardless of what the input turns out to need. An agentic system is different in one load-bearing way: the sequence of steps isn’t fixed in advance. The system itself, usually via a language model reasoning in a loop, decides what to do next based on what’s happened so far, which tool to call, whether it has enough information to answer, or whether it needs to go gather more first.

That single shift — from a fixed sequence to a decided sequence — is what “agentic” means, and it’s also exactly what makes these systems harder to design well than a normal pipeline. A fixed pipeline fails predictably, in the same place, the same way, every time a particular kind of input breaks it, which makes it straightforward to test and debug. An agentic system can fail in a combinatorial number of ways, because the path it takes through its available actions varies from run to run even on similar inputs. Designing an agentic system well is largely about constraining that variability just enough to keep the system reliable, without constraining it so much that you’ve quietly rebuilt a fixed pipeline and lost the flexibility that justified using an agent in the first place. Everything that follows — architecture, tools, memory, guardrails, failure handling, observability, and testing — is really one long answer to that same balancing problem, viewed from a different angle each time.

Deciding whether the problem needs an agent

Given how much harder agentic systems are to build reliably than fixed pipelines, the first design decision has to happen before any architecture gets drawn at all: does this problem require an agent, or would a simpler, deterministic pipeline solve it just as well with far less complexity and far more predictability? This question gets skipped more often than it should, usually because agentic architectures are exciting to build, and that excitement quietly overrides a more sober assessment of what the task demands.

A useful test is to ask whether the number and order of steps needed to solve a task is knowable in advance, for essentially all inputs the system will realistically see. If a task always follows the same sequence — fetch the record, validate it, format a response — a fixed pipeline will be more reliable, easier to debug, and cheaper to run than an agent making the same decisions dynamically at inference time for no benefit. Agentic design earns its complexity specifically when the number of steps, their order, or which tools are relevant can’t be known ahead of time — when the task requires the system to investigate, adapt its approach based on intermediate findings, or recover from an unexpected result by trying something different rather than failing outright. A customer support question that always resolves the same documented way doesn’t need an agent. A customer support question that might require checking three different systems, in an order that depends on what the first check reveals, is a much better candidate.

Choosing a single-agent or multi-agent architecture

Once a task has earned the complexity of an agentic approach, the next decision is architectural shape: should this be one agent handling the entire task, or several agents each responsible for a narrower piece of it, coordinated by some orchestration layer. This decision matters more than it might first appear, because it determines how much reasoning burden sits on any single component and how failures propagate through the system.

A single agent, given a broad enough set of tools and a capable enough model, can handle a surprisingly wide range of tasks, and it has an advantage in simplicity: there’s one reasoning loop, one place decisions get made, one context to reason about when something goes wrong. The trade-off is that a single agent’s context window and reasoning capacity become a shared resource across everything it’s asked to do, and as the number of available tools and the complexity of the task grows, that single agent’s decision-making tends to get noisier — it has more options to weigh at every step, and more opportunities to pick a plausible-looking but wrong path.

Multi-agent architectures address this by dividing responsibility: a research agent that only searches and reads, a planning agent that only decides what to do next, a writing agent that only produces final output, coordinated by a supervisor agent or a defined handoff protocol between them. This division tends to make each individual agent’s job narrower and therefore more reliable — a research agent with access only to search tools makes fewer kinds of mistakes than one generalist agent juggling search, writing, and calculation all at once. The cost is coordination complexity: agents now need a way to hand off work to each other, share relevant context without flooding each other with irrelevant detail, and recover gracefully when one agent’s output isn’t what the next agent expected. As a rule of thumb, start with a single, well-scoped agent for as long as it stays reliable, and split into multiple agents specifically when you can point to a concrete way a single agent’s reasoning is degrading under the combined weight of too many responsibilities — not because a multi-agent architecture sounds more sophisticated. Starting simple and splitting only when an observed problem justifies it also keeps the coordination overhead proportional to actual need, rather than paying multi-agent complexity costs upfront for a benefit that may never materialize for a task.

Designing the reasoning and planning loop

Whichever architectural shape you land on, every individual agent needs a defined loop it runs to decide what to do at each step — this is the mechanism that produces the “decided rather than fixed” behavior that makes the system agentic in the first place. The most common pattern, often called ReAct, alternates between the agent reasoning in natural language about what it currently knows and what it should do next, and then taking an action based on that reasoning — a tool call, a search, a calculation — observing the result, and feeding that result back into the next round of reasoning. This loop continues until the agent decides it has enough information to produce a final answer, or until some limit — a maximum number of steps, a time budget — forces it to stop.

Designing this loop well means being deliberate about a few things. The reasoning step needs enough of the right context to make a good decision — the original task, what’s been tried so far, what worked and what didn’t — without being so bloated with irrelevant history that the agent’s attention gets diluted across too much text, which is a measurable degradation in how well language models use long contexts. The stopping condition needs to be explicit rather than left entirely to the model’s judgment about when it’s “done,” because a poorly defined stopping condition is one of the most common sources of agents that either quit too early, before solving the problem, or loop far longer than necessary, burning time and cost re-trying variations of an approach that isn’t working. And the loop needs a hard ceiling — a maximum number of steps or a timeout — as a backstop, because even a well-designed stopping condition can fail to trigger in an edge case, and an agent looping indefinitely is a failure mode you want bounded by design, not caught by accident.

Designing the tool interface and action space

An agent’s reasoning loop is only as good as the actions it’s able to take, which raises the next design question directly: what tools does this agent have access to, and how are they defined. It’s tempting to give an agent as many tools as possible on the theory that more capability can only help, but this instinct works against reliability in a predictable way — every additional tool is another option the agent has to correctly choose between at every single reasoning step, and more options generally means more opportunities to pick the wrong one, especially when several tools have overlapping or unclear purposes.

The tools that work best in practice are narrowly scoped, clearly named, and accompanied by a description precise enough that the model can reliably tell, from the task at hand, whether a tool is the right one to reach for. A tool called “search” that vaguely searches “everything” is harder for an agent to use well than three separate, clearly distinguished tools — one for searching documentation, one for searching a customer database, one for searching recent support tickets — because the separation itself does some of the reasoning work for the agent, narrowing down which tool is relevant before the model even has to weigh subtler signals. Tool outputs matter just as much as tool definitions: a tool that returns a huge, unstructured blob of raw data forces the agent to spend its reasoning capacity parsing that output before it can act on it, where a tool that returns a clean, structured, appropriately-sized result lets the agent move directly to deciding its next step. Designing good tools is, in large part, designing good interfaces — the same discipline that makes a well-designed API pleasant for a human developer to use tends to make the same API easier for an agent to use correctly too.

Designing for cost and latency, not just correctness

A reasoning loop that calls a language model at every step, potentially several times before a task is done, raises a practical concern that’s easy to defer until it’s already a problem: every one of those calls costs money and takes time, and an agentic system that’s technically correct but too slow or too expensive to run at the volume it needs to handle isn’t a usable design, however well it performs in isolated testing. Designing for cost and latency alongside correctness, rather than as an afterthought bolted on once a system already works, tends to produce meaningfully better outcomes than optimizing purely for capability first and trying to make it fast and cheap later.

A few concrete levers matter here. Model selection doesn’t have to be uniform across an entire system — a smaller, faster, cheaper model is often perfectly capable of handling routing decisions or simple tool-selection steps, reserving a larger, more expensive model specifically for the steps that need its stronger reasoning, such as synthesizing a final answer from complex retrieved information. This kind of model cascading can cut cost substantially without a proportional hit to quality, precisely because not every step in an agentic loop demands the same level of reasoning capability. Caching is another meaningful lever: if a system frequently encounters similar sub-tasks or tool calls, caching results for identical or near-identical calls avoids redundant model calls and redundant tool execution entirely, which compounds significantly at usage volume even though it’s easy to overlook while a system is still small enough that redundant calls don’t yet show up as a cost.

Parallelization matters too, in cases where a task’s steps don’t depend on each other’s results — if an agent needs to check three independent data sources to answer a question, running those three tool calls concurrently rather than sequentially can cut wall-clock latency substantially, even though it doesn’t reduce the total amount of model or tool work being done. And step limits, already discussed as a safety backstop against runaway loops, do double duty as a cost control: a hard ceiling on how many reasoning steps a task is allowed to take bounds the worst-case cost of any single run, which matters considerably more once a system is handling production volume than it does during early testing on a handful of example tasks. None of these levers should be treated as optional polish applied after a system is proven to work — the shape of an agentic system’s cost and latency profile is set largely by architectural decisions made early, and retrofitting efficiency onto a system designed without it in mind is considerably harder than designing with it in mind from the start.

Designing memory: what the agent carries forward

As an agent works through a multi-step task, it accumulates a growing history of what it’s tried, what it found, and what it decided — and how that history gets managed is its significant design decision, distinct from the tools and the reasoning loop, because naive handling of it is one of the most common sources of both cost blowups and quality degradation in agentic systems. The simplest approach — just appending every reasoning step, every tool call, and every observation to a single growing context and passing all of it back into the model at every step — works for short tasks and breaks down as task length grows, both because context windows have limits and because models attend less reliably to information buried in the middle of a very long context, a well-documented effect that directly undermines an agent’s ability to make good decisions late in a long task.

Better-designed systems distinguish between different kinds of memory serving different purposes. Working memory holds what’s directly relevant to the current step — the immediate task, the most recent few observations — and stays intentionally small. Task memory holds a compressed summary of everything accomplished so far in the current run, updated as the task progresses rather than left to grow unbounded, so the agent retains the gist of its progress without carrying every raw detail. Longer-term memory, when a system needs it, persists facts or preferences across separate runs entirely — what a particular user prefers, what worked well on a similar task last time — stored and retrieved deliberately rather than simply accumulated in context. Designing memory well means deciding, for each piece of information an agent encounters, which of these categories it belongs to, and building a deliberate process for compressing or discarding what’s no longer needed rather than treating every prior message as permanently relevant.

Building in guardrails and verification

Because an agentic system’s path through a task isn’t fixed in advance, it’s also capable of taking actions nobody explicitly anticipated — which is exactly why guardrails need to be a first-class part of the design, not a feature bolted on after something goes wrong in production. Guardrails operate at a few different levels, and a well-designed system layers several of them rather than relying on just one.

At the narrowest level, individual tools can enforce their constraints — a tool that modifies data can require confirmation for destructive operations, or refuse to act outside a defined scope entirely, regardless of what the agent’s reasoning concluded. At a broader level, the system can validate an agent’s proposed action before it executes, checking it against a set of rules — does this action stay within the user’s permissions, does it match the kind of action this agent is meant to be taking — and blocking or flagging anything that doesn’t. For higher-stakes actions specifically, a human-in-the-loop checkpoint, where the agent proposes an action and waits for explicit approval before executing it, trades away some autonomy in exchange for a safety margin, and is often the right design choice for actions that are expensive, irreversible, or externally visible, even in a system that’s otherwise fully autonomous for lower-stakes steps. The overarching design principle is to match the level of oversight to the actual cost of a mistake at each step, rather than applying the same level of caution — too little or too much — uniformly across an entire system regardless of what’s at stake at any moment.

Planning for failure instead of assuming success

Guardrails reduce how often an agent takes a bad action, but they don’t eliminate failure altogether, which means the next design question is what the system does when something does go wrong — a tool call errors out, a piece of retrieved information turns out to be wrong, an intermediate step produces a result the agent can’t make sense of. Systems designed without an explicit answer to this question tend to fail in the least graceful way possible: either the whole task crashes on the first error, or the agent silently continues with an assumption that’s wrong, producing a confidently incorrect final result with no visible sign that anything went sideways.

A better-designed system treats failure as an expected, ordinary event that needs its explicit handling, not an edge case handled by accident. That can mean retrying a failed tool call with adjusted parameters, falling back to an alternative tool or approach when the first one doesn’t work, or — critically — recognizing when a task can’t be completed with the information and tools available, and reporting that honestly rather than generating a plausible-sounding answer anyway. This last point matters more than it might seem: an agentic system that never says “I couldn’t complete this” is not more capable than one that sometimes does — it has simply hidden its failures behind confident-sounding output, which is a worse outcome for anyone relying on the system than a visible, honest failure would have been.

Observability: knowing whether it’s working

Every design decision covered so far — architecture, tools, memory, guardrails, failure handling — only pays off if there’s a reliable way to tell whether the resulting system is working well in practice, which is what makes observability a design requirement from the start rather than something added once problems start showing up. Because an agentic system’s path through a task varies from run to run, understanding why a particular run succeeded or failed requires being able to see the actual sequence of reasoning, tool calls, and observations that run took — not just its final output.

This means logging each step of the reasoning loop in a structured, reviewable way: what the agent decided to do, why, what tool it called, what came back, and how that fed into its next decision. With that trace available, a failure can be diagnosed — was it a bad initial plan, a tool that returned unexpected data, a reasoning step that misinterpreted a correct observation — rather than just observed as an unexplained wrong answer. At a system level, this same tracing data supports the kind of ongoing evaluation that catches quality drift before it becomes a visible problem: tracking how often tasks complete successfully, how many steps they typically take, where in the loop failures cluster, and how those numbers shift over time as the underlying data, tools, or usage patterns change. Building this instrumentation in from the start is considerably cheaper than retrofitting it onto a system that’s already in production and already generating failures nobody can fully explain.

Testing an agentic system before it reaches production

The same variability that makes observability necessary once a system is live makes testing it before launch a different exercise than testing a deterministic pipeline, and it’s worth designing that testing approach deliberately rather than assuming the same techniques that worked for simpler software will transfer unchanged. A deterministic pipeline can be tested with a fixed set of inputs and exact expected outputs, because the same input always produces the same output. An agentic system, by design, doesn’t offer that guarantee — the same task can legitimately be solved via slightly different paths on different runs, which means testing needs to evaluate whether the outcome and the reasoning that produced it were sound, not whether the exact sequence of steps matched some predetermined script.

In practice, this means building a test suite around representative task scenarios rather than exact input-output pairs — a set of realistic situations the system needs to handle well, each with a way to judge whether a run’s final outcome and the path it took to get there were acceptable, even if that path varies between runs. It also means deliberately testing the edges the system is expected to handle gracefully, not just the cases where everything goes right: what happens when a tool call fails, when required information isn’t available, when a user’s request is ambiguous enough that the right first move is to ask a clarifying question rather than guess. A system that’s only ever been tested against clean, cooperative scenarios will reveal its actual failure modes for the first time in production, which is a considerably more expensive place to discover them than a test suite built with those edge cases in mind from the start. Testing an agentic system well is less about confirming it produces an output and more about confirming it makes sound decisions across the range of situations it will encounter — which is a different discipline than testing conventional software, and one worth building deliberately rather than improvising once the system is already live.

Orchestration patterns for multi-agent systems

For systems that did end up needing multiple agents rather than one, a further design question follows naturally: how do those agents coordinate with each other. A few patterns show up repeatedly in practice, each suited to a different shape of problem. In a supervisor pattern, one central agent owns the overall task and delegates sub-tasks to specialist agents, receiving their results back and deciding what happens next — this pattern keeps overall control centralized and relatively easy to reason about, at the cost of making the supervisor itself a potential bottleneck and single point of failure. In a sequential handoff pattern, agents pass work to each other in a defined sequence, each one completing its piece before handing off to the next — straightforward to build and debug, but less flexible when a task doesn’t naturally decompose into a fixed sequence of stages.

In a more decentralized mesh pattern, agents communicate more directly with each other as peers rather than through a single central coordinator, which offers more flexibility for complex, non-linear tasks but introduces coordination challenges — without careful design, it becomes harder to reason about the overall system’s behavior, and failure modes multiply as the number of possible interaction paths between agents grows. As with the single-versus-multi-agent decision earlier, the right orchestration pattern follows from the actual shape of the problem rather than from an abstract preference for one architecture over another: a task with a fixed high-level sequence of stages fits a sequential handoff well, a task that needs central quality control and a single source of truth for decisions fits a supervisor pattern well, and a task where sub-problems interact with and inform each other in ways that can’t be predicted in advance is one of the few cases where the added complexity of a mesh pattern is worth its cost.

Bringing it together: a worked example

These decisions are easier to see clearly when walked through against one concrete case rather than left as abstract principles, so consider a support triage system designed to take an incoming customer message, figure out what’s being asked, gather whatever information is needed to answer it, and either resolve it directly or route it to the right team with useful context attached. Applying the design questions in order: does this need an agent at all? Yes, in a meaningful way — the right sequence of checks depends on what the message is about, and can’t be predicted from the message alone, since a billing question and a bug report require checking entirely different systems in a different order.

Architecture: this fits a supervisor pattern reasonably well — one triage agent that classifies the incoming message and coordinates, delegating to a billing-lookup specialist for account and payment questions and a technical-diagnostics specialist for product issues, rather than one generalist agent trying to be equally good at both. The reasoning loop for the triage agent is intentionally short: classify, delegate, synthesize the specialist’s result into a response — deliberately not an open-ended loop, because the task doesn’t need one at this top level, even though the specialist agents underneath it may run their longer loops when the situation calls for deeper investigation. Tools are scoped narrowly per agent — the billing specialist gets access to the billing system and nothing else, the diagnostics specialist gets access to product logs and known-issue databases and nothing else — so that neither agent is ever choosing between tools that don’t apply to its actual job.

Memory here stays deliberately light: each specialist agent works within the scope of the single incoming request, without needing to carry forward unrelated history from other conversations, though the system as a whole might maintain longer-term memory of a customer’s past issues, retrieved deliberately when relevant rather than always present in context. Guardrails matter most around actions with consequences — issuing a refund, closing an account — which route through an explicit confirmation step rather than executing automatically, while lower-stakes actions like looking up an order status execute directly without added friction. Failure handling means that if the billing specialist can’t find an account record at all, the system doesn’t fabricate an answer — it says so explicitly and routes to a human agent with the attempted lookup details attached, rather than returning a synthesized-sounding answer that has no basis. And observability means every classification decision, every specialist delegation, and every tool call in the process gets logged in a way that lets a team later see exactly why a ticket got routed where it did, which is what makes it possible to improve the system’s classification accuracy over time rather than guessing at what’s going wrong from user complaints alone.

Common mistakes that show up once systems reach production

Across all of these decisions, a consistent set of mistakes shows up repeatedly once agentic systems move from a controlled demo into messy production usage, and naming them directly is more useful than leaving them to be discovered independently by every team that builds one of these systems. The most common is scope creep in the tool set: a system that launches with three well-defined, narrowly scoped tools gradually accumulates more as new use cases come up, until the agent is choosing between a dozen overlapping options and making worse decisions than it did with three — a sign that tools need periodic pruning and consolidation, not just addition.

A second common mistake is under-investing in the stopping condition and step limit until an agent gets stuck in an expensive, unproductive loop in production, at which point it becomes an urgent fix rather than a design decision made calmly in advance. A third is treating guardrails as a one-time addition rather than something that needs to evolve alongside the system — new failure modes surface as usage grows, and a guardrail set frozen at launch tends to miss exactly the failure modes that only show up once users start using the system in ways the original design didn’t anticipate. And a fourth, perhaps the most consequential, is skipping observability early because the system “seems to be working” in initial testing, only to find later that there’s no way to diagnose why quality has quietly degraded, because nobody can see the actual reasoning traces behind the outputs that are now going wrong. Every one of these mistakes traces back to the same root cause: treating agent design as a one-time architectural decision rather than as an ongoing discipline that has to keep pace with how the system is used once it’s live.

That ongoing discipline is really the throughline connecting every decision covered here. Confirming an agent is needed, choosing an architecture that matches the problem’s shape, keeping tools narrow and well-defined, managing memory deliberately instead of letting it grow unchecked, layering guardrails proportional to the cost of a mistake, planning for failure as an expected event rather than an exception, building in observability from day one, and testing against realistic, messy scenarios rather than clean ones — none of these are one-time boxes to check off during an initial build. They’re commitments a team keeps making as the system encounters more of the real world than any design process could have anticipated in advance, which is exactly what makes agentic systems different from the more predictable software most teams are used to building, and exactly what makes designing them well a distinct skill worth developing deliberately.