What is AI native stack?

Summary slides

“Stack” is already a familiar, well-worn term in software engineering generally — the combination of languages, frameworks, databases, and infrastructure a team chooses to build and run a system — and the AI native stack is that same concept applied to the technology choices specific to building around AI. Because this space has matured quickly and the available tooling keeps expanding, it helps to organize the stack by the functional category each piece serves, tracing roughly the same structure as the architectural layers discussed elsewhere in this knowledge base, while focusing here specifically on the concrete tools and vendor categories available for each one, and on how a team goes about choosing between them well.

The model layer: choosing among foundation model providers

At the base of nearly every AI native stack sits a choice, or more often several choices, of which underlying models will provide the system’s actual reasoning capability. This category includes major foundation model providers offering general-purpose models accessible through an API, as well as a growing ecosystem of open-weight models a team can host and run themselves rather than calling out to an external provider. The practical tradeoffs here run along a few consistent dimensions: API-based providers typically offer the most capable models with the least operational burden, since the provider handles the infrastructure, scaling, and ongoing model improvements, but come with per-request cost, data governance considerations around sending information to an external service, and dependency on that provider’s availability and pricing decisions. Self-hosted open-weight models shift that balance the other way — more operational responsibility and upfront infrastructure investment, in exchange for tighter control over cost at high volume, data residency, and independence from any single external provider’s decisions.

Most mature AI native stacks don’t settle on a single model for every task, but instead combine several: a highly capable model for complex reasoning, a smaller and faster model for simple, well-defined sub-tasks like classification or extraction, and sometimes a specialized model fine-tuned for a narrow, high-volume task where a general-purpose model would be needlessly expensive. Choosing well at this layer means resisting the temptation to standardize on a single model for everything simply because it’s the most capable one available, and instead matching each task in the system to the model whose capability, latency, and cost profile fits that task’s requirements — the multi-model approach discussed in the related article on AI native architecture.

The data and retrieval layer: vector databases, hybrid search, and knowledge infrastructure

The data and knowledge layer discussed in the related architecture article maps, in practice, onto a category of tooling built around storing and retrieving information in a form a model can reason over well. This category includes dedicated vector databases, purpose-built for storing and efficiently searching the kind of numerical representations that let a system find semantically relevant content rather than only content matching exact keywords; hybrid search systems that combine traditional keyword search with this kind of semantic search, often producing more reliable retrieval than either approach alone; and increasingly, vector search capability added directly to existing, more traditional databases, letting a team add retrieval capability to data that’s already stored in a familiar system rather than standing up an entirely separate database just for this purpose.

Choosing among these options depends heavily on a team’s existing data infrastructure and the scale of what they’re building. A team with a large, existing investment in a particular traditional database, and a comparatively modest retrieval need, often does better adding vector search capability to that existing system than introducing an entirely new, separate database to manage. A team building a system where retrieval quality and scale are central to the product’s core value, however, more often benefits from a dedicated, purpose-built vector or hybrid search system, since these systems are typically optimized specifically for the retrieval performance and scale that a general-purpose database, even with vector capability added, wasn’t originally designed to prioritize.

The ingestion and pipeline layer: keeping the knowledge base current

Sitting alongside the retrieval database itself is a category of tooling responsible for the ingestion pipeline discussed in the related architecture article — pulling information in from its original sources, processing it into a retrievable form, and keeping that representation current as the underlying information changes. This category includes document processing tools that handle extracting usable text and structure from varied source formats; chunking and embedding tooling that breaks source content into retrievable pieces at a sensible granularity and converts them into the representation a retrieval system searches over; and increasingly, managed data pipeline platforms that handle the ongoing orchestration of keeping this whole process running reliably as new content arrives and existing content changes.

This is a category many teams underinvest in relative to its actual importance, often because it’s less visible and less immediately demonstrable than the model or interface layers, echoing the mistake discussed in the related architecture article about building visible layers before the foundational ones underneath them. A well-chosen retrieval database delivers only as much value as the ingestion pipeline feeding it provides in terms of coverage, structure, and freshness, and teams that treat this category as an afterthought, cobbled together with ad hoc scripts rather than maintained infrastructure, tend to find their retrieval quality plateauing well below what their chosen database is capable of supporting.

The orchestration layer: frameworks for coordinating multi-step reasoning and tool use

The orchestration and reasoning layer discussed in the related architecture article maps onto a category of frameworks specifically built to help a team implement the orchestrator-worker, tool-use, and progressive context-loading patterns discussed in the related design-patterns article, without building all of that coordination logic entirely from scratch. This category ranges from lightweight libraries that handle the mechanics of formatting tool definitions and parsing structured model output, to more comprehensive frameworks that provide a full set of abstractions for building multi-step, multi-agent workflows, including built-in support for memory, state management across a session, and pre-built integrations with common external tools and services.

The tradeoff at this layer tends to run between speed of initial development and long-term flexibility. A comprehensive framework can get a team to a working system considerably faster, especially for teams newer to building this kind of orchestration logic, but can also impose its opinions about how a system should be structured, which occasionally conflict with what a more unusual use case needs. Teams building something that fits well within a framework’s intended use cases tend to benefit substantially from adopting one; teams building something unusual sometimes find it easier, in the long run, to build a thinner, more custom orchestration layer that gives them full control over exactly how requests move through the system, even at the cost of more initial development work.

The evaluation and observability layer: measuring and monitoring system behavior

The cross-cutting observability and evaluation layer discussed in the related architecture article maps onto a growing category of dedicated tooling built specifically for the unique measurement challenges AI-native systems present — challenges that traditional software monitoring tools, built around deterministic pass/fail checks, weren’t designed to handle well. This category includes evaluation platforms that let a team define and run test suites measuring output quality against a representative set of cases, tracking how that quality changes as prompts, retrieval logic, or underlying models change over time; tracing and logging tools built specifically to capture the kind of structured record of a request’s full path through retrieval, model calls, and tool use that the observability layer depends on; and increasingly, tools specifically built to detect model behavior drift — cases where a system’s real-world behavior on production traffic starts diverging from what evaluation on a fixed test set would predict.

This is a category that has matured rapidly precisely because so many teams, in the earlier years of AI-native system building, discovered the hard way just how costly it is to operate a system without it — the mistake, discussed repeatedly and consistently throughout this knowledge base, of treating evaluation as a nice-to-have addition rather than core infrastructure. Teams building an AI-native system today have considerably more mature, purpose-built options in this category than were available even a couple of years earlier, and choosing to adopt this kind of dedicated tooling early, rather than deferring it, tends to pay for itself many times over once a system reaches production scale and the cost of debugging without it becomes apparent.

The guardrail and safety layer: constraining what a system can output or do

Closely related to but distinct from evaluation is a category of tooling specifically focused on runtime guardrails — checks applied to actual production requests and responses, in real time, to catch and block or correct categories of problematic output before they reach a user or trigger a downstream action, as discussed in the related design-patterns article. This category includes content filtering and moderation tools that check outputs against defined policies before they’re delivered; structured validation tooling that verifies a model’s output conforms to whatever schema a downstream system depends on, discussed in the related design-patterns article as the structured-output pattern; and increasingly, more sophisticated tools specifically built to detect and block attempts to manipulate a system’s behavior through crafted input, a growing concern as AI-native systems take on more autonomous action with consequences.

The maturity of this category varies more than most others in the AI native stack, partly because the guardrails a system needs depend heavily on its domain and stakes — a low-stakes internal tool needs considerably less guardrail infrastructure than a customer-facing system authorized to take financial or legal action — and partly because this remains an actively evolving area as new categories of risk get identified through real-world experience. Teams building in higher-stakes domains tend to need to combine several guardrail tools rather than relying on a single one, layering content checks, structural validation, and action-constraints together, in the same way a well-built traditional security posture layers multiple independent defenses rather than relying on any single control alone.

The agent framework layer: building autonomous, tool-using systems

For systems building toward the autonomous agent type discussed in the related article on AI native systems, a further category of tooling has emerged specifically focused on the concerns unique to autonomous, multi-step, tool-using operation: managing longer-running tasks that may span multiple separate model calls and real-world actions over an extended period, handling the state and memory an agent needs to maintain coherent behavior across that extended operation, and providing the kind of structured checkpoint and approval mechanisms the human-in-the-loop pattern depends on for higher-stakes autonomous actions.

This category overlaps significantly with the broader orchestration layer discussed earlier, and the line between “an orchestration framework” and “an agent framework” is blurry in current tooling, with many frameworks supporting both simpler, more deterministic orchestration and autonomous, longer-running agent behavior within the same underlying toolkit. What distinguishes agent-tooling in practice is typically a stronger built-in emphasis on handling uncertainty and failure gracefully across a longer sequence of autonomous steps — since the graceful-degradation pattern discussed in the related design-patterns article matters more and is harder to get right, the longer and more autonomous a sequence of actions becomes without a human checking in along the way.

How to choose among the options within each category

Given how many options typically exist within each of these categories, and how quickly new ones continue to appear, it helps to have a consistent set of criteria for evaluating them, rather than choosing based on which option happens to be most visible or most recently released. The first criterion worth weighing heavily is how well a tool’s abstractions match the system’s actual complexity — a lightweight tool that handles a team’s needs well is usually a better choice than a more elaborate, feature-rich platform that solves problems the system doesn’t have, echoing the broader principle, discussed in the related article on AI native design patterns, of adding structure and complexity in proportion to need rather than by default.

The second criterion is how easily a tool lets a team swap out what sits behind its interface — a vector database that locks a team into a hard-to-migrate data format constrains future flexibility considerably more than one built around open, portable standards; an orchestration framework that deeply entangles a team’s logic with one model provider’s particular API constrains the model-layer swappability discussed in the related architecture article. Favoring tools that keep these interfaces clean and portable, even when a more tightly integrated alternative might be marginally faster to adopt initially, tends to pay off considerably as a system matures and its requirements inevitably shift.

The third criterion is the maturity and stability of the tool itself, weighed honestly against how critical the function it serves is to the system. For a foundational, hard-to-replace piece of infrastructure like the primary data store, favoring a mature, well-established option with a track record of reliability at scale is usually worth more than a newer alternative’s marginally more elegant interface. For a more peripheral, easily replaceable piece of tooling, being an early adopter of a promising newer option carries much less risk, since a poor choice there is comparatively cheap to correct later.

How the stack tends to evolve as a system moves through its lifecycle

The stack a team chooses for an early pilot, discussed in the related article on AI native systems’ lifecycle, is often appropriately different from the stack a mature, production-scale system eventually needs, and recognizing this as a natural, expected evolution rather than a sign of an earlier bad decision helps teams avoid both of two common failure modes: over-investing in production-grade infrastructure for an early pilot that hasn’t yet proven its underlying value, and under-investing in the infrastructure a system needs once usage and stakes have grown well beyond what its original, lighter-weight stack was ever designed to support.

An early pilot typically does well with the simplest, fastest-to-adopt option in each category — a managed, all-in-one platform that handles retrieval, orchestration, and evaluation together with minimal setup, even if that platform’s flexibility and cost-efficiency at scale are weaker than a more assembled, best-of-breed alternative would offer. As a system moves through the hardening stage and toward production maturity, teams often find themselves replacing pieces of that initial, convenient stack with more specialized, more scalable, or more cost-efficient alternatives in the categories where the pilot’s original choice is becoming a constraint — rarely replacing the entire stack at once, but incrementally swapping out individual pieces as each one’s limitations become the actual bottleneck worth addressing next. Designing the system’s architecture with the swappability discussed earlier in mind is precisely what makes this kind of incremental stack evolution possible without requiring a disruptive, ground-up rebuild each time one piece of the stack needs to change.

The interface and delivery layer: where the stack meets the actual user

The interface layer discussed in the related architecture article maps onto a further category of tooling, somewhat more familiar to traditional software teams than the layers discussed above, focused on delivering an AI-native system’s capability to its users. This category includes conversational interface frameworks specifically built to handle the streaming, incremental-response pattern discussed in the related design-patterns article well, managing the particular user-experience challenges of a response that arrives progressively rather than all at once; embeddable widget and component libraries that let a team add AI-native interaction to an existing product’s interface without rebuilding that interface from scratch; and API gateway and integration tooling that lets other internal or external systems consume the AI-native system’s capability programmatically, rather than only through a human-facing interface.

This layer is worth naming explicitly within the stack, even though it draws more heavily on familiar, general-purpose web and application development tooling than the more AI-categories discussed above, because the particular demands AI-native interaction places on an interface — handling streaming responses gracefully, surfacing the kind of transparency into retrieval and action-taking discussed in the related architecture article, managing the perceived latency of a multi-step reasoning process — differ enough from a typical traditional application’s interface needs that treating this layer as a simple extension of existing frontend tooling, without any AI-consideration, tends to produce an interface that undersells what the rest of the stack underneath it is capable of delivering.

How build-versus-buy decisions play out differently across the different layers of the stack

A recurring decision a team faces at nearly every layer of the AI native stack is whether to adopt an existing tool or platform in a category, or to build a custom solution specifically tailored to the system’s needs, and the right answer to this question varies considerably depending on which layer is in question, in ways worth being explicit about rather than applying a single blanket build-versus-buy philosophy uniformly across the entire stack.

At the model layer, building your foundation model from scratch is, for the overwhelming majority of teams, simply not a realistic option given the scale of investment required, making this layer one where adopting an existing provider’s offering is close to universal, with the decision being which provider and which model rather than whether to build one internally at all. At the data and retrieval layer, adopting an existing vector or hybrid search database is similarly close to universal for the underlying storage and search technology itself, though the ingestion pipeline built around that database — the logic for how an organization’s particular data sources get processed and kept current — is almost always custom, because that logic depends too heavily on an organization’s data sources and structure for any off-the-shelf tool to handle it completely out of the box.

At the orchestration and agent framework layer, the build-versus-buy decision splits both ways depending on a system’s complexity, as discussed earlier — a system whose needs fit well within an existing framework’s abstractions benefits from adopting one, while a system with unusual requirements sometimes does better with custom-built orchestration logic, even though that means forgoing a framework’s built-in conveniences. At the evaluation and guardrail layer, teams increasingly adopt dedicated platforms for the underlying infrastructure — running test suites, capturing traces, applying content checks — while building custom logic for the criteria those platforms evaluate against, since generic evaluation criteria rarely capture what matters for a system’s domain and stakes without meaningful customization.

Recognizing this layer-by-layer variation, rather than applying a single organizational philosophy of “build everything ourselves” or “buy everything available” uniformly across the whole stack, tends to produce better outcomes than either extreme. Teams that reflexively build everything themselves often end up re-deriving solutions to problems the broader ecosystem has already solved well, at cost in time and engineering effort that could have gone toward the system’s differentiated parts instead. Teams that reflexively buy everything available sometimes end up with a stack of loosely integrated, individually excellent tools that don’t fit their system’s requirements as well as a more deliberate mix of adopted infrastructure and custom logic, tailored specifically to what makes their system’s needs different from a generic use case, would have provided.

How stack choices affect the ongoing cost of running an AI native system

Beyond initial development speed and long-term flexibility, the tools chosen at each layer of the stack have a direct, often underappreciated effect on a system’s ongoing operational cost, and this dimension deserves its explicit consideration rather than being treated as a secondary concern that only matters once a system reaches unusually large scale. At the model layer, cost varies enormously depending on which model handles which task, which is exactly why the multi-model approach discussed earlier matters as much for cost management as for raw capability — routing simple, high-volume requests to a smaller, less expensive model while reserving a larger, more expensive model for complex reasoning can reduce a system’s total model cost substantially without meaningfully affecting the quality of what users experience, provided the routing logic deciding which request goes where is itself well designed.

At the data and retrieval layer, cost is driven heavily by the volume of data being indexed and the frequency with which it needs to be kept current, and teams that index far more content than their retrieval logic benefits from using — including large volumes of low-value or rarely relevant content simply because it was easy to ingest — often pay considerably more for storage and search infrastructure than a more deliberately curated, higher-quality, smaller index would cost, while frequently getting worse retrieval quality in the bargain, since a bloated index makes it harder for a retrieval system to surface the most relevant content among a larger pool of less relevant alternatives. At the orchestration layer, cost is driven by how many separate model calls and retrieval operations a typical request triggers, which is why the earlier point about not over-applying the orchestrator-worker pattern to tasks that don’t need that level of decomposition matters as much for cost as it does for latency and reliability — every additional step in an orchestrated pipeline is an additional cost, on top of whatever value it adds.

Building cost visibility into the observability layer discussed earlier — tracking not just whether a system’s outputs are correct but what each request costs to serve, broken down by which layer and which tool or model contributed to that cost — is what makes it possible to make these tradeoffs deliberately rather than discovering, only once a monthly infrastructure bill arrives unexpectedly large, that an architectural or tooling choice was costing far more than its actual contribution to system quality justified. Teams that build this kind of cost observability in alongside their quality observability from the start tend to catch cost inefficiencies early, while they’re still cheap and easy to correct, rather than after they’ve compounded across months of production traffic into an expensive problem to unwind.

How organizational scale changes which stack choices make sense

The right stack for a small team building a single AI-native system looks meaningfully different from the right stack for a larger organization building and maintaining several AI-native systems across different teams simultaneously, and this organizational dimension deserves separate consideration from the system-level factors — scale, stakes, existing infrastructure — discussed earlier, because it introduces considerations that only emerge once more than one system and more than one team are involved.

A single small team building one system benefits most from optimizing each stack choice specifically for that one system’s requirements, without much need to consider whether the same choice would also serve some other, different system well, since no other system’s requirements are in play. A larger organization building multiple AI-native systems across different teams faces a different calculation: standardizing on a shared stack across systems, even when a more specifically optimized choice exists for any single one of those systems individually, often produces better organization-wide outcomes than letting each team choose independently, because a shared stack lets the organization build deep expertise, shared tooling, and shared operational practices around a smaller number of technologies, rather than spreading that expertise thin across many different, individual team choices that each require their own specialized knowledge to operate and maintain well.

This tension — between what’s locally optimal for any single system and what’s organizationally optimal across many systems together — doesn’t have a single universally correct resolution, but organizations that navigate it well tend to share a few practices in common. They typically standardize deliberately on a smaller number of layers where the organizational benefit of consistency clearly outweighs the cost of any individual system giving up its locally optimal choice — often the data and retrieval layer and the evaluation and observability layer, since expertise and shared tooling there tend to compound particularly well across systems — while leaving more room for individual teams to choose independently at layers where a system’s requirements are more likely to diverge from what other systems in the organization need, such as the orchestration and agent framework layer, where the right level of complexity and the right abstractions vary considerably from one system’s use case to another’s.

Common mistakes teams make when assembling an AI native stack

The single most common mistake, observed across teams at nearly every stage of maturity, is over-indexing on the model layer when choosing a stack — spending disproportionate time and attention selecting the single most capable foundation model currently available, while giving comparatively little thought to the data, retrieval, evaluation, and guardrail tooling that surrounds it. Given how much of a system’s real-world quality depends on the layers surrounding the model, discussed throughout this knowledge base’s coverage of AI native architecture and principles, this imbalance tends to produce systems whose actual, measured quality falls well short of what the underlying model’s raw capability would suggest is achievable, because the surrounding stack was never given the same level of deliberate selection and investment.

A second mistake, common especially among teams newer to this space, is adopting an all-in-one platform that bundles every category together, attracted by the convenience of a single integrated solution, without checking whether that platform’s implementation of each individual category is competitive with dedicated, best-of-breed alternatives available separately. All-in-one platforms serve certain teams and certain stages of a system’s lifecycle well, particularly early on, but teams that adopt one at production scale without ever benchmarking its individual components against the alternatives sometimes discover, once the system is deeply dependent on that platform, that a dedicated alternative would have served one or more categories considerably better, at which point migrating away from the bundled dependency is often considerably more work than choosing more deliberately from the start would have required.

A third mistake, and one that tends to go unnoticed the longest, is neglecting the swappability criterion discussed above in favor of whichever option is fastest to get working today, accumulating tight, hard-to-reverse dependencies on tools across every layer of the stack without any deliberate consideration of how difficult each of those dependencies will be to unwind later if the tool turns out to be the wrong long-term choice. This mistake compounds over time, because each additional tightly coupled dependency makes every future stack change more disruptive than the last, until a team finds itself unable to adopt an obviously better alternative in any single category without a disruptive, costly rework of the entire surrounding system.

A fourth mistake, easy to overlook until it’s already expensive, is neglecting cost observability specifically, treating cost as something to review only when a surprising bill arrives rather than as a metric tracked continuously alongside quality from the very start, which tends to let inefficient choices at the model, data, or orchestration layer compound quietly for months before anyone notices they’re driving disproportionate expense relative to the value they contribute to the system.

A fifth mistake, specific to larger organizations running several systems at once, is either forcing rigid stack standardization across every system and every layer regardless of how different those systems’ requirements are, or allowing complete stack fragmentation with no shared standards anywhere, rather than finding the more deliberate middle path discussed above — standardizing where shared expertise and tooling compound across systems, while leaving room for individual teams to diverge in the places where their systems’ actual requirements clearly call for it.

What connects all five of these mistakes is choosing pieces of the stack in isolation, based on each piece’s individual appeal, rather than evaluating the whole assembled stack against the system’s actual, whole-system requirements — coverage across every functional category the architecture needs flexibility to evolve as the system matures, proportionate investment matched to how consequential each layer is to the system’s overall quality and reliability, and honest visibility into what each choice costs to run over time. Teams that evaluate their stack this way, deliberately and as a coherent whole rather than category by category in isolation, tend to build AI native systems on a foundation that can support the system’s growth over years, rather than one that has to be substantially rebuilt the first time the system’s requirements outgrow the convenient choices that got it off the ground quickly.