How do you design low latency AI systems?

Summary slides

Latency has always mattered in software generally, throughout the entire history of the discipline, but it matters in a considerably more visible way for AI-native systems in particular, because the underlying model inference step frequently takes long enough on its own — hundreds of milliseconds to several seconds, depending on the model and the task — that it becomes the dominant, most noticeable component of a request’s total response time, in a way a typical traditional software operation’s near-instant execution rarely was. Designing well for low latency in this context means understanding where that particular time goes in a request’s lifecycle, and which of several distinct, purpose-built techniques address each source of delay most directly and most effectively.

Why perceived latency and actual latency are different design targets

A request’s actual total latency — the time from when a user submits a request to when the complete response is fully available — is one meaningful measure, but it isn’t the same as perceived latency, which is how quickly a user subjectively feels the system responded, and the two can differ substantially depending on how a response is delivered. A response that takes a full eight seconds to fully generate but starts streaming visible text within roughly half a second feels considerably more responsive and attentive to a user than a response that takes that exact same eight seconds but shows absolutely nothing at all until the complete text is finally ready, even though both cases carry precisely identical total, actual latency by any objective measure.

This distinction matters because streaming, discussed further below in its dedicated section, is often the single highest-leverage latency technique available to a team precisely because it improves perceived latency dramatically without necessarily improving the system’s actual total latency at all — a valuable, comparatively low-cost intervention for exactly the reason that user experience depends far more on perceived responsiveness in the moment than on any raw, actual-latency number a user never directly observes, measures, or even becomes aware of themselves. Designing for low latency well means treating perceived and actual latency as two related but distinct targets, each addressed by a partially different set of techniques, rather than assuming that reducing actual total latency is the only lever available for improving how responsive a system feels to the people using it.

How streaming reduces perceived latency by showing progress as it happens

Streaming itself, echoing the pattern discussed throughout this knowledge base’s coverage of AI native design patterns, delivers a model’s output incrementally as it’s generated rather than waiting for the entire complete response before showing anything at all, letting a user start reading a response well before it’s finished. This is particularly, specifically valuable for AI systems precisely because model-generated text often takes long enough to fully complete that showing absolutely nothing during that entire generation window would feel noticeably unresponsive to whoever is waiting on it, in a way that would be far less noticeable, and far less consequential, for a typical traditional software operation completing quietly in a few tens of milliseconds or less.

Building streaming well requires more than just technically enabling it at the model API level — it requires designing the surrounding interface to take advantage of incremental delivery, handling partial, in-progress content gracefully rather than presenting a jarring, incomplete-looking display while generation is still underway. It also interacts meaningfully with the verification and guardrail discipline discussed throughout this knowledge base’s coverage of AI native design patterns, since a guardrail check that needs to evaluate a complete response before releasing any of it to the user necessarily delays the streaming benefit until that check finishes, which means systems combining streaming with guardrails need a deliberate design decision about whether to stream unchecked and guard retroactively, or check first and stream only after, each carrying a different latency-versus-safety tradeoff worth making explicitly rather than by default.

How minimizing sequential model calls reduces the latency that compounds across a multi-step pipeline

Every model call in a sequential chain adds its latency on top of every call before it, which means a workflow built around the orchestrator-worker pattern discussed throughout this knowledge base’s coverage of AI native design patterns can accumulate substantial total latency if its steps run strictly one after another, even when each individual step is reasonably fast on its own. A five-step sequential pipeline, each step taking a modest one second, totals five full seconds of latency purely from the sequential structure, regardless of how fast any single step is in isolation.

Reducing this compounding latency means examining a pipeline’s actual dependency structure specifically for steps that don’t need to wait for each other and running those independent steps in parallel rather than sequentially, echoing the same complexity-discipline discussed in the related article on AI native design patterns but applied here specifically to latency rather than to reliability or cost. It also means questioning, for each step in a sequential chain, whether that step is necessary at all for the request at hand, since a pipeline built to handle a system’s most complex possible case by default sometimes runs every one of its steps even for a simpler request that could have skipped several of them safely — building conditional logic that skips unnecessary steps for simpler requests reduces latency specifically for the requests that don’t need the full pipeline’s complete sequence to be run in full.

How model size selection trades raw capability against response speed

A larger, more capable model generally takes longer to produce a response than a smaller one, echoing the multi-model routing discussed throughout this knowledge base’s coverage of AI native architecture and stack, but applied here specifically to latency rather than to cost — the same routing decision that saves money by sending a simple task to a smaller model also, as a direct side effect, tends to reduce latency for that task, since a smaller model typically responds faster in addition to costing less. This means the routing discussed in the related article on cost-efficient AI system design frequently serves double duty as a latency-reduction technique as well, though the two goals don’t align perfectly in every case worth naming directly.

The cases where they diverge matter for latency-sensitive design: a task might be simple enough, in terms of the reasoning it requires, to route to a smaller model purely for cost reasons, but if that smaller model, for whatever reason takes longer to respond than a larger one for that particular kind of request — which does happen for certain model and task combinations — then the cost-optimal routing choice and the latency-optimal routing choice diverge, and a system with strict latency requirements needs to weigh actual measured response time specifically, not just cost, when deciding which model handles a latency-sensitive step.

How caching serves latency in a way distinct from how it serves cost

Caching, discussed in the related article on cost-efficient AI system design primarily as a technique for avoiding repeated model-call expense, serves an equally direct latency benefit worth naming in its own right: a cached result returns essentially instantly, skipping the entire model inference delay altogether, which makes caching one of the few techniques in this article that improves both cost and latency simultaneously without the tradeoff most other techniques in this article carry between the two.

This dual benefit makes caching worth prioritizing specifically for any request pattern that recurs frequently and predictably, echoing the same semantic-matching discussion covered in the related article on cost-efficient AI system design, since the latency improvement compounds with the cost improvement for exactly the same cached requests rather than requiring separate investment to capture each benefit independently. Systems with a latency-sensitive user experience — an interactive assistant a user expects to respond quickly, as opposed to a background batch process where latency matters far less — benefit disproportionately from investing in caching specifically because of this dual latency-and-cost payoff, relative to systems where latency isn’t a primary concern and caching’s benefit is purely about cost.

How batching, a cost-efficiency win, directly works against latency

Batching, discussed in the related article on cost-efficient AI system design as a technique for using compute more efficiently by processing several requests together, directly trades against latency in a way worth stating plainly: accumulating a batch requires waiting for enough requests to arrive before processing them together, which adds delay to each individual request in that batch compared to processing it immediately and alone. This is precisely why batching is the right choice specifically for workloads that can tolerate that added delay and the wrong choice for latency-sensitive, real-time interactive workloads.

Recognizing this direct tension explicitly is what lets a system apply batching correctly rather than either applying it uniformly, which would degrade a latency-sensitive interactive experience unacceptably, or avoiding it uniformly, which would leave cost savings unclaimed for the parts of a workload that could have tolerated the tradeoff perfectly well. A well-designed system routes latency-sensitive requests through an unbatched, immediate-processing path even at higher per-request cost, while routing latency-tolerant background work through a batched path to capture the cost benefit where the corresponding latency cost doesn’t matter.

How network and infrastructure placement affects latency independent of model choice

Beyond the model-level and architectural techniques discussed throughout this article, a meaningful share of a request’s total latency in many systems comes from network transit time — the time spent sending a request to wherever the underlying model runs and receiving the response back — which is a separate latency source from the model’s inference time and worth addressing through its distinct set of techniques. A system whose users are geographically distant from wherever its underlying infrastructure is physically located pays a network latency cost on every single request purely from that physical distance, regardless of how fast the model itself responds once a request reaches it.

Addressing this latency source typically means choosing infrastructure regions close to where a system’s actual users are located, or in some cases running inference at multiple geographic locations and routing each request to whichever location is closest to that request’s origin, echoing the distributed infrastructure discussion in the related article on AI native versus cloud native architecture. This is a separate design lever from every other technique discussed in this article, addressing a different source of delay, and a system that’s optimized every model-level and architectural technique discussed elsewhere in this article while still running its infrastructure far from its actual users leaves this particular, often substantial source of latency completely unaddressed.

How time-to-first-token differs from total generation time as its distinct metric worth measuring separately

Beyond the perceived-versus-actual latency distinction discussed earlier in this article, it’s worth measuring and optimizing two separate latency metrics explicitly: time-to-first-token, the delay before any output at all begins appearing, and total generation time, the delay before the complete response finishes. These two metrics don’t always move together, and a system optimized for one without attention to the other can end up with a lopsided latency profile that undersells the actual user experience improvement available.

A system with a fast time-to-first-token but a slow total generation time still benefits substantially from streaming, since a user sees progress almost immediately even though the full response takes a while to complete — this is a good outcome for perceived responsiveness even without any improvement to total actual latency. A system with a slow time-to-first-token, by contrast, feels sluggish and unresponsive from the very first moment regardless of how quickly it generates once it starts, which means investing specifically in reducing whatever causes that initial delay — a slow retrieval step that has to complete before generation can even begin, an expensive pre-processing step, unnecessary setup overhead — often yields a more noticeable improvement in perceived responsiveness than an equivalent investment in speeding up the later, already-streaming portion of generation. Teams that measure and track time-to-first-token as its explicit metric, separate from total generation time, tend to identify and address the bottleneck that’s hurting perceived responsiveness most rather than optimizing generation speed broadly without knowing which phase of the request’s lifecycle is the dominant source of felt delay.

How orchestration overhead itself contributes latency separate from the model calls it coordinates

Beyond the latency of the model calls themselves, discussed throughout most of this article, the orchestration logic coordinating those calls — parsing a model’s structured output, deciding which step to run next, formatting the input for the following step — adds its latency on top of the model calls it coordinates, echoing the orchestrator-worker pattern discussed throughout this knowledge base’s coverage of AI native design patterns. This orchestration overhead is often small relative to any single model call’s latency, but it compounds across a multi-step pipeline in the same way the sequential model-call latency discussed earlier in this article does, and a pipeline with many steps can accumulate meaningful orchestration overhead even if every individual orchestration step is fast in isolation.

Reducing this overhead means keeping orchestration logic itself lightweight and efficient — avoiding unnecessary data transformation or serialization between steps, minimizing redundant validation that could be consolidated into a single pass rather than repeated at every step — and periodically profiling a pipeline’s actual latency breakdown specifically to check how much of its total latency traces to orchestration overhead versus the model calls themselves, since a pipeline where orchestration overhead turns out to be a surprisingly large share of total latency calls for a different optimization approach than a pipeline where nearly all the latency traces to the model calls, and a team that assumes orchestration overhead is always negligible without measuring it can end up optimizing model calls indefinitely while leaving a measurable latency source in the orchestration layer completely unaddressed.

How speculative execution can reduce latency for the common case at some cost for the uncommon one

A more advanced latency technique, echoing the progressive context-loading pattern discussed throughout this knowledge base’s coverage of AI native design patterns, is speculative execution: starting work on a likely next step before the current step has fully confirmed that next step is the right one, so that if the guess turns out correct, the system has already made progress and doesn’t pay that step’s full latency sequentially after the guess is confirmed. This trades some wasted compute for the cases where the speculation turns out wrong against a latency benefit for the cases where it turns out right, and it’s worth applying specifically where a system has enough confidence in what’s likely to happen next that the speculation succeeds often enough to justify its cost.

Building this well requires a calibrated sense of how often a speculative guess turns out correct for a system’s request patterns, since speculation applied to an unpredictable next step wastes compute on failed speculation far more often than it saves latency on successful speculation, while speculation applied to a next step that’s highly predictable captures a meaningful latency benefit at a comparatively modest wasted-compute cost. This technique tends to be worth the added implementation complexity specifically for systems with high request volume and predictable patterns, where even a modest per-request latency improvement compounds into a meaningful aggregate benefit, rather than for lower-volume or unpredictable systems where the added complexity isn’t likely to pay for itself.

How the tension between latency and accuracy needs the same calibrated judgment cost decisions require

Echoing the calibrated, stakes-proportional thinking discussed throughout this knowledge base’s coverage of AI native principles and applied to cost decisions in the related article on cost-efficient AI system design, latency optimization carries its version of the same tension: a technique that reduces latency by using a smaller model, skipping a verification step, or truncating context can also reduce the accuracy or reliability of the resulting output, and treating every latency reduction as an unconditional win without weighing this tension risks shipping a faster but worse system for tasks where that accuracy reduction matters.

Applying this calibration well means identifying which parts of a system are latency-critical, in the sense that a user’s actual experience depends heavily on fast response time, versus which parts have more headroom to prioritize accuracy over speed because the task’s actual stakes warrant that priority — a quick, exploratory interaction reasonably tolerates a faster, slightly less capable model in exchange for responsiveness, while a request whose output will directly inform a significant decision reasonably tolerates a few extra seconds of latency in exchange for the added reliability a more careful process provides. Systems designed with this calibration in mind apply the most aggressive latency optimization specifically where responsiveness matters most and the accuracy cost is acceptable, while preserving more careful, less latency-optimized handling for the smaller share of requests where getting the answer right matters considerably more than getting it back quickly.

How latency requirements should shape architecture decisions from the very start rather than being addressed after the fact

Because several of the techniques discussed throughout this article — parallelizing pipeline steps, choosing models specifically for response speed, designing for streaming — touch a system’s fundamental architecture rather than being easily retrofitted adjustments, latency is a design dimension best addressed from a project’s very beginning rather than optimized after a system is already built around an architecture that wasn’t designed with latency requirements in mind at all. A system built as a strictly sequential pipeline from the start, without early attention to which steps could have run in parallel, often requires a disruptive architectural rework to parallelize later, considerably more costly than designing the parallel structure in from the beginning would have been.

Teams that identify their system’s actual latency requirements early — how fast does a kind of interaction need to feel to the people using it, and which parts of the system are on that latency-critical path — and design the architecture around those requirements from the start tend to reach a low-latency system considerably more efficiently than teams that build first and attempt to retrofit latency optimization onto an architecture that was never designed with those requirements in mind, discovering the architecture’s fundamental sequential or otherwise latency-unfriendly structure only once users start experiencing the resulting sluggishness directly and the pressure to fix it arrives well after the easier, cheaper opportunity to design it in from the start has already passed.

How to measure latency in a way that reflects what users experience

Given how many distinct latency sources this article has covered, it’s worth closing with a practical note on measurement, since a system’s latency profile is hard to improve deliberately without measuring it in a way that reflects what users experience, rather than a single aggregate average that can obscure exactly the kind of variation that matters most for a responsive system. Average latency alone is a particularly misleading metric for AI systems specifically, because a distribution with a low average but a long tail of occasionally very slow requests can still leave a meaningful share of users experiencing poor responsiveness, even while the aggregate average number looks perfectly healthy on a dashboard.

Teams that measure latency well track percentile-based metrics specifically — the latency experienced by the slowest five or ten percent of requests, not just the average — because this tail behavior is often what determines whether users perceive a system as reliably responsive or as occasionally, frustratingly slow in a way that colors their overall impression of the product regardless of how fast the typical request is. This measurement also needs to be broken down by the request types and pipeline paths discussed throughout this article, echoing the granular cost observability discussed in the related article on designing cost-efficient AI systems, since a single aggregate latency number, even a percentile-based one, doesn’t reveal which kind of request or which step in a pipeline is responsible for the latency that matters most to address. A team that builds this kind of granular, percentile-aware latency observability early tends to identify and address its most consequential latency problems considerably faster and more precisely than a team relying on a single, coarse average number that reveals a system is “somewhat slow” without ever revealing exactly why, or exactly for whom, or exactly under what conditions that slowness occurs.

How latency requirements differ across different categories of AI-native application

The latency bar a system needs to clear varies considerably depending on what kind of application it serves, and treating every AI system as needing the same aggressive latency target regardless of its actual use case leads to over-investing in latency optimization where it doesn’t matter and under-investing where it critically does. A real-time conversational interface, echoing the interaction-model discussion in the related article on AI native applications, needs the aggressive streaming and parallelization techniques discussed throughout this article, because a user actively waiting in a live conversation notices and is frustrated by even a modest delay in a way that directly and immediately shapes their experience of the product.

A background batch-processing workload, by contrast, tolerates considerably higher latency per individual request, since nobody is actively watching and waiting for any single result in real time, which means the aggressive, often more complex and more expensive latency techniques discussed throughout this article are frequently not worth their implementation cost for this category of workload at all, and the cost-efficiency techniques discussed in the related article on designing cost-efficient AI systems, including the batching this article’s discussion identified as directly trading against latency, become the more appropriate priority instead. A system serving both categories of workload — some real-time, some tolerant of delay — benefits from recognizing this distinction explicitly and applying a different latency-versus-cost balance to each category deliberately, rather than either over-investing in latency optimization for the batch workload that never needed it, or under-investing in latency optimization for the real-time workload that does.

Common mistakes teams make when designing for low-latency AI systems

The single most common mistake, observed across teams building interactive AI systems at nearly every stage of maturity, is failing to implement streaming for interactive use cases, discussed at considerable length earlier in this article, leaving a user staring at a blank or loading screen for the system’s entire actual generation time, when incremental, streamed delivery would have made that exact same actual latency feel considerably more responsive to that user, at a comparatively modest implementation cost relative to the perceived-responsiveness benefit it reliably provides.

A second mistake, often made simply because a pipeline’s steps were originally written in the same order a person would naturally think through the underlying task, is building a sequential pipeline without ever examining which of its steps could run in parallel, discussed in detail above, accumulating compounding latency from a strictly sequential structure that a closer, more deliberate look at the pipeline’s actual dependencies would have revealed wasn’t necessary for a meaningful share of its individual steps at all.

A third mistake, easy to fall into once a team has already seen cost savings from batching in one part of a system, is applying batching indiscriminately across an entire system’s workload without carefully distinguishing latency-tolerant background work from latency-sensitive interactive requests, discussed at considerable length above, degrading the interactive experience noticeably for the sake of a cost benefit that a more selective, workload-application of batching would have captured just as fully, without ever incurring that unnecessary latency cost on the requests that needed to stay fast.

A fourth mistake, one that can leave a team puzzled about why their carefully optimized system still feels slow to users, is optimizing every model-level and architectural latency technique discussed throughout this article while entirely neglecting network and infrastructure placement, discussed at length above, leaving a substantial and comparatively easily addressable source of latency completely unaddressed, simply because it sits outside the model-focused techniques a team may have already spent considerable time and effort optimizing everywhere else in the system.

A fifth mistake, subtle enough that it often survives well past initial launch without anyone specifically noticing it, is measuring only total generation time without ever tracking time-to-first-token as its separate distinct metric, discussed at considerable length earlier in this article, optimizing generation speed broadly without ever identifying whether the true bottleneck hurting perceived responsiveness most is really the generation phase at all, rather than whatever delay happens before the very first token even begins streaming to the user.

A sixth mistake, one that tends to persist precisely because it’s rarely the first place anyone looks, is assuming orchestration overhead in a multi-step pipeline is always negligible without ever deliberately measuring it, discussed in detail above, optimizing model calls indefinitely while leaving a measurable, and sometimes surprisingly large latency source sitting entirely unaddressed within the coordination logic that connects those individual calls together into a coherent whole.

A seventh mistake, which can go wrong in either of two opposite directions depending on which way a team happens to err, is applying speculative execution indiscriminately across a system, or alternatively avoiding it entirely without ever seriously evaluating whether a system’s actual request patterns are predictable enough to benefit from it, discussed at considerable length above, either wasting meaningful compute on speculation that rarely pays off, or leaving a steadily compounding latency benefit unclaimed for a system whose patterns were predictable enough all along to have made the technique worthwhile.

An eighth mistake, easy to make when the team celebrating a latency win isn’t the same team that later notices the resulting quality regression, is treating every latency reduction as an unconditional, unambiguous win without ever weighing the tension against accuracy discussed at considerable length above, shipping a system that’s faster but measurably worse for exactly the tasks where the accuracy cost of that added speed mattered considerably more than the latency improvement itself was ever really worth.

A ninth and final mistake, and arguably the most expensive of all nine to correct once it’s discovered, is deferring latency considerations until well after a system is already built around an architecture that was never designed with any latency requirements in mind, discussed in detail above, facing a disruptive and costly architectural rework to retrofit parallelization or streaming behavior that would have been considerably cheaper, faster, and altogether easier to design in correctly from the very earliest days of the project.

What ultimately connects all nine of these mistakes, considered carefully together as a whole rather than as nine separate, unrelated missteps each with its isolated cause, is treating latency as a single, undifferentiated problem to be solved with a single favored technique, rather than recognizing that different sources of delay — generation time, first-token delay, sequential pipeline structure, orchestration overhead, batching-induced queueing, network transit, and the underlying architecture itself — each require their distinct technique and their explicit measurement to address well, and that several of these techniques directly trade against the cost-efficiency goals discussed in the related article on designing cost-efficient AI systems and against the accuracy and reliability goals discussed throughout this knowledge base’s broader coverage of AI native principles. Teams that identify which latency sources matter most for their particular system, measure each one explicitly and separately rather than relying on a single coarse aggregate number, and apply the technique that addresses each one deliberately and appropriately rather than reaching reflexively for a single favored technique across every situation regardless of fit, tend to build systems that feel consistently responsive to their actual real-world users without sacrificing more cost efficiency or reliability than each latency improvement honestly justifies in return.