How do you design scalable AI applications?

Summary slides

Scalability in traditional software has a comparatively well-understood shape: add more servers, and a stateless application generally handles more traffic in something close to direct proportion. AI applications complicate this familiar picture in an important way worth naming directly, because the underlying model call itself, sitting at the center of nearly every request, isn’t something a team can simply provision more of on its own infrastructure the way it can spin up more application servers — that call typically depends on an external provider’s capacity and rate limits, which means scaling an AI application well requires addressing constraints outside the team’s infrastructure, not only within it.

Why model-provider rate limits are often the first scaling bottleneck a growing system hits

A traditional application’s scaling bottleneck is usually somewhere within the team’s infrastructure — database connections, raw server capacity, network bandwidth — all of which a team can typically provision more of on its own, given enough budget, enough lead time, and enough advance planning to order and deploy that additional capacity before it’s needed. An AI application’s first scaling bottleneck, by contrast, is frequently sitting entirely outside the team’s infrastructure altogether: the underlying model provider’s rate limits, which cap how many requests or how much total token volume an account can send within a period, regardless of how much the team’s application infrastructure could otherwise handle if it were the only constraint in play.

Managing this bottleneck well means treating it as a first-class scaling concern from quite early in a system’s development, rather than discovering it only once unplanned production growth triggers it for the first time, usually at the worst and least convenient possible moment for a team to be addressing it reactively for the first time. This typically means building request queuing that smooths sudden bursts of demand into a steadier rate the provider’s limits can sustain over time, rather than simply sending requests as fast as they happen to arrive and accepting whatever errors result once that limit is inevitably exceeded; negotiating higher rate limits directly with a provider as demonstrated volume steadily grows over time, since most providers are willing to offer meaningfully higher limits to established customers who can point to a demonstrated usage history rather than only a projected future need; and, for systems with high, sustained volume, deliberately spreading load across multiple separate provider accounts or across multiple different model providers entirely, echoing the swappability principle discussed throughout this knowledge base’s coverage of AI native architecture, so that no single account’s individual rate limit ends up capping the entire system’s total achievable throughput on its own.

How keeping the orchestration layer stateless enables horizontal scaling

The orchestration layer discussed throughout this knowledge base’s coverage of AI native architecture — the component coordinating a request’s path through retrieval, model calls, and tool use — needs to be stateless to scale horizontally the way traditional web application servers typically do, meaning any instance of that layer can handle any incoming request without depending on state held only in memory on one instance. A system that accidentally accumulates request-state in a single orchestration instance — an in-memory conversation history, a partially completed multi-step process tracked only in that instance’s local memory — can’t simply add more instances to handle more load, because a request routed to a different instance than the one holding its relevant state won’t have access to the context it needs.

Building statelessness means externalizing any state a request’s processing depends on into a shared, external store — a database or cache accessible to every orchestration instance equally — rather than holding it in a single instance’s local memory. This is a familiar discipline from traditional distributed systems engineering, but it’s worth naming explicitly for AI applications specifically because the memory-and-personalization pattern discussed throughout this knowledge base’s coverage of AI native design patterns can tempt a team toward exactly the kind of instance-local state that breaks horizontal scaling, if that pattern isn’t deliberately implemented against a shared, external store from the start.

How the data and retrieval layer scales differently from the model-calling layer

The data and retrieval layer discussed throughout this knowledge base’s coverage of AI native architecture faces its distinct scaling considerations, separate from the model-provider rate-limit and orchestration-statelessness concerns discussed above. As a system’s data volume grows — more documents, more accumulated interaction history — the retrieval layer’s search performance can degrade if the underlying index and search infrastructure weren’t built to scale with that growing volume, independent of how many requests per second the system is handling.

This means a scalable AI application needs to think about two related but distinct scaling dimensions at once: scaling to handle a growing number of concurrent requests, which touches the orchestration and model-calling layers primarily, and scaling to handle a growing volume of underlying data, which touches the retrieval layer’s indexing and search infrastructure primarily, and treating these two dimensions as a single, undifferentiated “scaling problem” tends to obscure exactly which investment a system’s current bottleneck calls for. A system that scales its request-handling capacity well while neglecting its retrieval layer’s ability to scale with growing data volume can still hit a bottleneck as its underlying knowledge base grows, even if its request-handling capacity alone would have supported considerably higher concurrent load. Building for both dimensions deliberately, rather than assuming that scaling request handling automatically scales data handling as well, is what lets a system grow along both axes without either becoming an unaddressed limiting factor.

How graceful degradation under capacity limits differs from ordinary error handling

Even a well-scaled system eventually reaches a capacity limit under sufficiently high demand, and how a system behaves at that moment — echoing the graceful-degradation pattern discussed throughout this knowledge base’s coverage of AI native design patterns, applied here specifically to capacity limits rather than to model-output uncertainty — matters considerably for how that moment is experienced by users. A system that simply fails unpredictably once capacity is exceeded, returning inconsistent errors or silently dropping requests, produces a considerably worse experience than one that fails predictably and communicates clearly what’s happening.

Building this well means defining explicit, carefully considered behavior for the capacity-exceeded case well before it’s ever reached for the first time in production, rather than improvising a response only once users are already experiencing whatever unplanned behavior the system happens to fall back on by default: queuing excess requests with a clear, honest estimate of wait time rather than rejecting them outright, when a brief queue is tolerable for the use case at hand; shedding load deliberately and predictably, prioritizing important requests over less critical ones when true capacity limits are reached, rather than treating every request as equally important once resources become scarce; and communicating capacity issues clearly to users rather than presenting a confusing, generic error that gives no indication of what’s happening or when normal service might resume. Systems that design this behavior deliberately, before production demand ever forces the issue, handle capacity crunches considerably more gracefully than systems that only discover their actual behavior under load once an unplanned surge in demand reveals it for the first time.

How load testing for AI applications needs to account for token throughput, not just request count

Traditional load testing typically measures a system’s capacity in requests per second, which is a reasonable proxy for traditional software where each request’s processing cost is comparatively uniform. AI applications complicate this measure because a request’s actual cost and processing time can vary enormously depending on how much context it includes and how much output it generates, which means a system’s capacity is better measured in token throughput — the actual volume of text being processed and generated — rather than in a simple request count that doesn’t capture this underlying variability.

A load test that only measures how many requests per second a system handles, without accounting for the token volume each of those requests carries, can produce a misleadingly optimistic capacity estimate if the test’s request mix happens to be lighter, in terms of token volume, than a system’s production traffic is. Building load tests that specifically model realistic token volume — using representative context lengths and output lengths rather than artificially short test requests that complete faster than production requests ever would — produces a considerably more accurate picture of a system’s actual scaling limits than a request-count-only load test ever could. This matters especially for systems whose request mix varies in complexity across different use cases, since a load test built around the system’s simplest, lightest-weight requests will systematically overstate capacity for any production traffic mix that includes a meaningful share of heavier, more context-intensive requests, and a team that only discovers this gap once varied production traffic arrives has effectively been testing against a scenario its actual users were never going to produce in the first place.

How autoscaling infrastructure needs to account for AI-resource demands

Traditional autoscaling — automatically adding or removing server instances based on measured load — assumes a fairly uniform, predictable relationship between request volume and the compute resources each request consumes, which holds reasonably well for traditional web applications but breaks down somewhat for AI applications, echoing the inference-infrastructure discussion in the related article on AI native versus cloud native architecture. A request triggering a single, small model call and a request triggering a complex, multi-step orchestrated workflow with several model calls consume different amounts of the underlying compute and, more importantly different amounts of external model-provider capacity, even if both count identically as a single “request” in a traditional autoscaling metric based purely on request count.

Building autoscaling that reflects an AI application’s resource demands means scaling based on metrics that capture this variability — queue depth for pending model calls, actual token throughput, or the rate at which the orchestration layer is consuming available provider capacity — rather than relying purely on traditional metrics like CPU utilization or raw request count, which can significantly understate or overstate an AI application’s resource pressure depending on the mix of request types arriving at any moment. Teams that build autoscaling around these AI-metrics tend to provision capacity considerably more accurately, avoiding both the wasted cost of over-provisioning based on a misleadingly low traditional metric and the availability risk of under-provisioning based on a metric that doesn’t capture where the resource pressure is coming from.

How database and vector-store scaling introduces its considerations beyond general retrieval-layer scaling

Beyond the general retrieval-layer scaling discussed earlier in this article, the underlying data stores an AI application depends on — traditional databases holding structured application data, and vector or hybrid search stores holding the retrieval-oriented content discussed throughout this knowledge base’s coverage of AI native stack — each carry their own scaling characteristics worth understanding separately rather than treating “the data layer” as a single, undifferentiated scaling concern.

Traditional databases scale through well-established techniques — read replicas, sharding, connection pooling — that most engineering teams already have practical experience with from building traditional software. Vector and hybrid search stores, by contrast, often have their distinct scaling characteristics specific to the particular underlying search technology a team has chosen, and these characteristics can behave quite differently from traditional database scaling in ways a team without prior experience in this technology might not anticipate — search latency and accuracy can both degrade in ways that aren’t always straightforwardly predictable as the underlying index grows to a large scale, and the appropriate scaling response, whether that’s a different indexing strategy, additional search infrastructure, or a fundamentally different underlying technology altogether, depends on the store a team has chosen. Teams building a scalable AI application benefit from testing their vector or hybrid search store’s actual scaling behavior directly, at a data volume representative of where the application is expected to grow, rather than assuming its scaling characteristics will simply mirror a traditional database’s more familiar, well-understood behavior.

How multi-region deployment introduces complexity specific to AI applications

Deploying an AI application across multiple geographic regions, echoing the latency benefits of regional infrastructure placement discussed in the related article on designing low-latency AI systems, introduces its scaling complexity worth addressing directly. A traditional multi-region deployment typically replicates application logic and data across regions in a comparatively well-understood way; an AI application’s multi-region deployment additionally needs to account for whether the underlying model provider itself offers comparable capacity, latency, and even model behavior consistently across every region a team wants to serve from, since not every provider offers identical capability and capacity in every geographic region equally.

This means a scalable multi-region AI application needs to verify, specifically and directly, that its chosen model provider or providers can support the regions it intends to serve from at the capacity and latency the application requires, rather than assuming that a multi-region deployment strategy that worked well for traditional application logic will automatically extend cleanly to the model-calling layer as well. Teams that skip this verification sometimes discover, only once they’ve already built out a full multi-region deployment, that a region’s actual model access is more limited, more expensive, or meaningfully slower than the region they originally built and tested against, forcing either a costly redesign or an uncomfortable compromise in what the application can promise to deliver consistently across every region it claims to serve.

How organizational scaling — from prototype to many teams building on shared infrastructure — introduces its distinct challenges

Beyond the purely technical scaling considerations discussed throughout this article, an important dimension of scaling an AI application is organizational: as a system grows from a single team’s prototype into shared infrastructure that multiple teams across a larger organization build on top of, echoing the internal-platform discussion in the related article on AI native platforms, new challenges emerge that a purely technical scaling strategy doesn’t address on its own. Multiple teams sharing the same underlying model-provider capacity and rate limits, discussed earlier in this article, need some form of deliberate capacity allocation or prioritization, since a single team’s unexpectedly heavy usage can otherwise silently consume shared capacity that other teams were also depending on, without any of those other teams necessarily understanding why their unrelated requests suddenly started failing or slowing down.

Building this organizational scaling well typically means establishing capacity governance — allocating a defined share of shared provider capacity to each team or use case, with clear visibility into actual usage against that allocation, echoing the granular cost observability discussed in the related article on designing cost-efficient AI systems but applied here specifically to capacity rather than cost — so that one team’s growth doesn’t silently degrade another team’s service without any clear, attributable cause. Organizations that scale technically well but skip this organizational dimension often find that their AI infrastructure’s actual reliability, from any single team’s perspective, depends unpredictably on what every other team sharing that same infrastructure happens to be doing at any moment, an uncomfortable position for any team trying to build a reliable product on top of infrastructure whose practical capacity it can’t see or control.

How caching interacts with scalability specifically, beyond its cost and latency benefits

Caching, discussed in the related articles on designing cost-efficient and low-latency AI systems primarily for its cost and latency benefits, carries a scalability benefit worth naming in its own right: a cache hit skips the model call entirely, which means it also skips consuming any of the model-provider rate-limit capacity discussed earlier in this article as the first bottleneck a growing system typically hits. This makes caching a scalability lever, not just a cost and latency one, since every cached request effectively increases a system’s usable capacity within a fixed rate limit by simply not needing to draw on that limit at all.

This scalability benefit compounds specifically with the kind of request patterns most likely to recur predictably — a system serving a large number of users asking substantially similar questions benefits disproportionately from caching’s scalability effect compared to a system where every request is uniquely different, since the former can serve a meaningful share of its total request volume from cache without ever touching the underlying rate limit at all, while the latter gets comparatively little scalability benefit from caching regardless of how well the caching itself is implemented. Teams building for scale benefit from assessing how much of their actual expected request volume is likely to be cacheable in this way, since that assessment directly informs how much of their overall scaling challenge caching can realistically be expected to absorb versus how much needs to be addressed through the rate-limit management, provider diversification, and infrastructure scaling discussed throughout the rest of this article.

How scaling considerations should shape the initial choice between building a custom stack and adopting a managed platform

Echoing the build-versus-adopt discussion in the related article on AI native platforms, a system’s actual expected scale is an important input into whether building a custom stack or adopting a managed platform makes more sense for a team, worth naming explicitly here because the scaling considerations discussed throughout this article play out differently depending on which path a team takes. A managed platform typically handles a meaningful share of the infrastructure-level scaling challenges discussed in this article — autoscaling, database and vector-store scaling, sometimes even a degree of rate-limit management across the platform’s aggregated customer base — as part of what the platform itself provides, which can let a smaller team reach production scale without building and operating this scaling infrastructure themselves from scratch.

A custom-built stack, by contrast, requires a team to address every one of the scaling considerations discussed throughout this article directly, which demands more upfront engineering investment but also provides considerably more control over exactly how each bottleneck gets addressed, control that becomes more valuable as a system’s actual scale grows large enough that a managed platform’s more generic, one-size-fits-all scaling approach starts to feel like a constraint rather than a convenience. Teams evaluating this choice well weigh their actual, realistic expected scale honestly against both paths’ tradeoffs, rather than defaulting to a custom stack out of a general preference for control regardless of whether the team’s actual scale warrants that additional ownership burden, or defaulting to a managed platform indefinitely even once scale has grown large enough that the platform’s inherent limitations have become a measurable constraint on the system’s further growth.

How to think about scaling a system that combines multiple AI-native application types with different scaling profiles

Many AI applications aren’t a single, uniform workload but a combination of the different application types discussed throughout this knowledge base — a retrieval-and-knowledge component, a copilot-style assistant, perhaps a more autonomous agentic capability — each of which, echoing the taxonomy discussed in the related article on AI native systems, carries a different scaling profile worth addressing separately rather than as a single, undifferentiated scaling problem. A retrieval-and-knowledge component’s scaling challenge centers heavily on the data and retrieval-layer scaling discussed earlier in this article, since its primary cost and latency driver is typically the retrieval step itself. A more autonomous agentic capability’s scaling challenge centers more heavily on the sequential, multi-step model-call latency and rate-limit consumption discussed throughout this article, since a single agentic task can consume considerably more of a system’s available model-call capacity than a single, simpler retrieval-based request ever would.

Recognizing this variation matters because a single, uniform scaling strategy applied identically across every part of a combined system tends to under-invest in whichever component’s actual scaling profile differs most from whatever profile the uniform strategy happened to be designed around. Teams building scalable combined systems tend to assess and address each component’s scaling profile somewhat independently — even while sharing common infrastructure, observability, and rate-limit management across all of them — rather than assuming a single scaling approach, tuned around one component’s characteristics, will automatically serve every other, structurally different component in the combined system equally well.

Common mistakes teams make when designing for AI application scalability

The single most common mistake, observed across teams building AI applications at nearly every stage of growth, is failing to plan for model-provider rate limits until a growing system hits them directly in production, discussed at considerable length earlier in this article, discovering the constraint only once user-facing errors start appearing in front of actual customers, rather than having proactively built the queuing, provider negotiation, or multi-provider strategy that would have addressed the limit well before it ever became a visible, embarrassing production problem affecting users.

A second mistake, particularly insidious because it can pass every test a team runs during initial development, is accidentally introducing instance-local state into the orchestration layer, discussed in detail above, building a system that appears to work correctly during initial, low-traffic testing where every request happens to land on the same instance, but that breaks in subtle hard-to-diagnose ways once horizontal scaling routes a request to a different instance than the one silently holding the context that request depended on all along.

A third mistake, easy to overlook precisely because request-handling capacity and data volume grow on independent, often unrelated timelines, is scaling request-handling capacity while entirely neglecting the retrieval layer’s separate ability to scale with growing underlying data volume, discussed at some length above, hitting an unaddressed bottleneck in search performance as an organization’s underlying knowledge base steadily grows over time, even while the system’s request-handling capacity alone would have comfortably supported considerably higher concurrent load than it was able to deliver once that separate, unaddressed retrieval bottleneck was reached.

A fourth mistake, quietly dangerous precisely because it produces a confident, reassuring-looking number rather than an obvious warning sign, is load testing with an unrealistic request mix that meaningfully understates production token volume, discussed at length above, producing a capacity estimate that looks entirely comfortable and safe during testing but that doesn’t hold up once heavier, more representative production traffic reveals the system’s true, considerably lower real-world capacity than the original test ever measured.

A fifth mistake, one that tends to feel like reasonable, standard infrastructure practice right up until it demonstrably isn’t, is autoscaling purely on traditional metrics like CPU utilization or raw request count, discussed at some length earlier in this article, provisioning capacity that either wastes cost on over-provisioning based on a misleadingly low traditional signal, or risks availability problems from under-provisioning based on a metric that never captured where the AI-resource pressure was coming from all along.

A sixth mistake, rooted in a reasonable but ultimately mistaken assumption carried over from years of traditional database experience, is assuming a vector or hybrid search store’s scaling behavior will simply mirror a traditional database’s more familiar, well-understood characteristics, discussed in detail above, only discovering at large data volume, often well after significant production dependence has already accumulated, that the underlying search technology a team happened to choose degrades in latency or in retrieval accuracy in ways its team’s traditional-database intuition never anticipated or specifically prepared for in advance.

A seventh mistake, expensive specifically because it tends to surface only after a large infrastructure investment has already been made, is building a full multi-region deployment without first, carefully verifying that the chosen model provider offers comparable capacity, latency, and consistent behavior across every single region the application intends to serve from, discussed at length above, discovering only after that full multi-region build-out is essentially complete that a region’s model access is meaningfully more limited, more expensive, or noticeably slower than the original region the system was designed and thoroughly tested against from the start.

An eighth mistake, specific to larger organizations, where multiple distinct teams share the same underlying AI infrastructure, is scaling the purely technical infrastructure well while entirely skipping the organizational capacity governance discussed at some length above, leaving each individual team’s actual service reliability unpredictably dependent on what every other team sharing that same underlying infrastructure happens to be doing at any moment, without any clear attributable visibility into why a team’s entirely unrelated requests suddenly started failing or slowing down for no apparent reason of their own.

A ninth mistake, which can go wrong in either of two opposite directions depending on which way a team’s assumptions happen to lean, is underestimating or overestimating how much of a system’s actual scaling challenge caching could realistically absorb, discussed at some length above, either underinvesting in caching for a workload whose request patterns were predictable enough to benefit substantially from it, or over-relying on caching as a primary scaling strategy for a workload whose requests are too varied and too individually unique for caching to meaningfully reduce rate-limit and infrastructure pressure in any material way.

A tenth mistake, often traceable back to whichever option simply felt more comfortable or familiar to the team at the time the original decision was made, is choosing between a custom stack and a managed platform without honestly, carefully weighing the system’s actual, realistic expected scale against each option’s tradeoffs, discussed at length above, either building unnecessary custom scaling infrastructure for a system whose actual scale never warranted that considerable investment, or staying committed to a managed platform well past the point where its inherent limitations had already become a clearly measurable constraint on the system’s further, continued growth.

An eleventh and final mistake, closing out this list but no less consequential for coming last, is applying a single, uniform scaling strategy uniformly across a system that combines multiple different AI-native application types, discussed at considerable length above, under-investing specifically in whichever component’s actual scaling profile differs most from whatever profile the uniform strategy happened to be originally designed and tuned around, rather than assessing and addressing each individual component’s distinct scaling characteristics somewhat independently, even while still sharing common infrastructure across all of them where that sharing makes sense.

What ultimately connects all eleven of these mistakes, considered together as a coherent whole rather than as eleven separate, unrelated failures, is treating AI application scalability as though it followed the exact same, familiar shape as traditional software scalability, addressed purely through provisioning more application servers and more database capacity, rather than recognizing the distinct AI-bottlenecks — external provider rate limits, retrieval-layer data scaling, token-volume-sensitive capacity, AI-autoscaling metrics, vector-store scaling behavior, multi-region provider consistency, organizational capacity governance, caching’s scalability role, and build-versus-platform tradeoffs — that AI applications specifically introduce across their full real-world scope, well beyond what a traditional scaling playbook was ever designed to address. Teams that address each of these distinct bottlenecks deliberately and specifically, rather than simply assuming a traditional scaling playbook transfers unchanged onto a different kind of system with different constraints, tend to build AI applications that scale to sustained production demand over the long run, without the unpleasant, costly surprise of hitting an unaddressed, unfamiliar bottleneck only once growth has already arrived, already accelerated, and already started causing visible, hard-to-reverse harm to users who were counting on the system to simply keep working as it always had.