What is RAG architecture?

Summary slides

The core components every RAG architecture needs

Whatever technology choices a team makes, a working RAG architecture needs to account for a consistent set of components, each responsible for a distinct part of the overall process. An ingestion pipeline takes raw source content — documents, tickets, wiki pages — and turns it into something searchable: chunking it into appropriately sized pieces, computing vector embeddings for each chunk, and writing both the content and its embedding into a vector store, along with whatever structured metadata the system needs for filtering later. A vector store holds those embeddings and supports efficient similarity search against them at scale, typically alongside the ability to filter by metadata and, in a well-built system, to combine that similarity search with traditional keyword matching in a single query.

A retrieval service sits in front of the vector store and handles the logic of turning an incoming question into an actual search — sometimes rewriting or expanding the query first, applying whatever filters the request requires, and in more sophisticated architectures, reranking the initial results before deciding what to pass along. An orchestration layer coordinates the overall request: it takes a user’s question, calls the retrieval service, assembles a prompt combining the question with whatever was retrieved, and calls a generation service — typically a language model API — to produce the final answer, sometimes making multiple rounds of this cycle if the architecture supports more advanced patterns like query decomposition or agentic retrieval rather than a single fixed pass through the pipeline. Each of these components has its distinct responsibility, and how cleanly that separation is maintained — versus components bleeding into each other’s concerns — is one of the more consequential architectural decisions a team makes early on, whether they realize they’re making it deliberately or simply by accumulated default.

Following data through the architecture end to end

With those components named, it’s worth tracing an actual request through all of them in sequence, because the handoffs between components are where a surprising share of architectural decisions and architectural problems live. A request begins when a user’s question arrives at the orchestration layer, which typically first decides whether any query rewriting is needed — turning a conversational, context-dependent question into a self-contained one, for instance — before passing it to the retrieval service.

The retrieval service converts that query into a vector using the same embedding model the ingestion pipeline used originally — a detail worth flagging explicitly, since a mismatch between the embedding model used at ingestion time and the one used at query time silently breaks retrieval quality in a way that’s easy to overlook — and sends that vector, along with any structured filters, to the vector store. The vector store returns a ranked set of candidate chunks, which may then pass through an optional reranking step before the retrieval service hands a final, trimmed set of chunks back to the orchestration layer. The orchestration layer assembles those chunks into a prompt alongside the original question and any relevant conversation history, sends that assembled prompt to the generation service, and receives back the model’s response, which it may post-process — extracting citations, checking for particular required elements — before returning it to the user. Every one of these handoffs is a place where data has to be correctly formatted, correctly matched in scope, vocabulary, and permissions between components, and correctly monitored, and an architecture that’s vague about exactly what crosses each of these boundaries tends to accumulate exactly the kind of subtle bugs that are hard to trace precisely because no single component owns the point where two of them meet, which is a large part of why the handoffs themselves, and not just the components on either side of them, deserve deliberate design attention.

Monolithic versus modular architecture

Having traced the logical flow, the next architectural decision is how tightly or loosely these components are built and deployed relative to each other — as a single, tightly integrated application, or as separate, independently deployable services. A monolithic RAG architecture runs ingestion, retrieval, orchestration, and often even the generation call all within one application codebase and one deployment unit. This has advantages early on: it’s simpler to build, simpler to reason about as a whole, and avoids the network overhead and coordination complexity of calling out to several separate services for what’s conceptually just one single, coherent request from the user’s point of view.

A modular architecture splits these components into separate services — a dedicated retrieval service, a separate ingestion pipeline running independently of the request-serving path, an orchestration layer that calls out to both — each independently deployable and independently scalable. This trades away some of the monolith’s simplicity for benefits that tend to matter more as a system grows: ingestion, which is often bursty and resource-intensive, can scale and fail independently of the request-serving retrieval path, which needs to stay fast and available regardless of what ingestion happens to be doing at any moment; different components can be scaled to match their actual load patterns rather than all scaling together as one unit; and a team can iterate on, say, the reranking logic inside the retrieval service without needing to redeploy the entire application. As a general pattern, starting monolithic and deliberately splitting out a component once it has a concrete, demonstrated need to scale or iterate independently — rather than defaulting to a fully modular architecture from day one, before any component has proven it needs that independence — tends to avoid paying microservice coordination costs before a system has grown enough to need them. The signal worth watching for isn’t a fixed size or traffic threshold, but a concrete pain point — ingestion load starting to visibly slow down live queries, or a retrieval-quality change requiring a full redeploy of unrelated orchestration code — at which point splitting that component out is a targeted fix for an observed problem rather than speculative architecture built ahead of any actual need for it.

The ingestion pipeline as its own subsystem

Whether monolithic or modular, the ingestion pipeline deserves architectural attention distinct from the request-serving path, because its performance characteristics and failure modes are different from anything that happens while answering a live user question. Ingestion is typically bursty — a large batch of new or updated documents arrives at once, rather than a smooth, predictable trickle — and it’s usually not remotely as latency-sensitive as a live, user-facing request is, since a document being searchable a few minutes after it was updated is usually fine even in an architecture built for real-time freshness, where the actual requirement is bounded, reasonable staleness, not literal instantaneous updates.

This difference in characteristics argues for architecting ingestion as a queue-based, asynchronous pipeline rather than a synchronous process directly in the request path: a change to a source document triggers a message onto a queue, and a separate set of workers processes that queue at whatever pace matches available capacity, re-chunking and re-embedding the changed content and writing the result into the vector store, all without ever touching or competing for resources with the live retrieval path that’s simultaneously serving user questions. This queue-based decoupling is what allows ingestion to absorb a sudden burst of updates — a bulk document import, a large content migration — without that burst degrading response times for users actively querying the system at the same moment, which is a risk in an architecture where ingestion and query-serving share the same compute and the same request-handling code path without any deliberate separation between them.

The retrieval layer: where hybrid search architecture lives

The retrieval service is worth its architectural attention because it’s typically where the most retrieval-quality-critical logic lives, and how it’s structured determines how much flexibility a team has to improve retrieval quality later without a disruptive rework. A well-architected retrieval layer treats the actual search mechanics — vector similarity, keyword matching, metadata filtering, and any reranking — as composable stages within one coherent service, rather than scattering that logic across the orchestration layer or, worse, across whatever application code happens to call the retrieval service.

This matters architecturally because retrieval logic tends to need iteration and tuning considerably more often than the rest of a RAG system — adjusting how keyword and vector scores are balanced, adding a new metadata filter, introducing a reranking step that wasn’t there before — and a retrieval layer that’s cleanly separated as its own service, with a stable, well-defined interface to the orchestration layer above it, lets a team make those changes without touching or redeploying anything else in the system. An architecture that instead spreads retrieval logic across multiple places, with the orchestration layer directly querying the vector store and applying its ad hoc filtering logic inline, tends to make even a small retrieval-quality improvement into a change that touches code far outside where the actual retrieval logic conceptually belongs, which slows down exactly the kind of iterative tuning that retrieval quality most depends on.

The orchestration layer and prompt assembly

Sitting above retrieval, the orchestration layer’s core architectural responsibility is prompt assembly — combining a user’s question, whatever was retrieved, conversation history, and any system-level instructions into the actual text sent to a language model — and this deserves more architectural care than it often gets, because prompt assembly bugs are a surprisingly common, and surprisingly invisible, source of RAG quality problems. A well-architected orchestration layer treats prompt assembly as its well-tested, isolated piece of logic, with clear, deliberate rules about how much retrieved content fits into a context budget, how conversation history gets truncated or summarized once it grows long, and what happens when retrieval returns nothing usable at all.

This last case deserves particular architectural attention: an orchestration layer needs an explicit, designed path for “retrieval didn’t find anything relevant,” rather than silently assembling a prompt with an empty or near-empty context section and letting the model do whatever it does with that — which, as covered elsewhere, tends to be a plausible-sounding, ungrounded guess rather than an honest admission that nothing relevant was found. Architecting this explicit fallback path, rather than treating it as an edge case that happens to be handled by whatever the model does by default, is a small design decision with an outsized effect on how the whole system behaves under exactly the conditions where its behavior matters most.

Caching: where it helps and where it introduces risk

As request volume grows, caching becomes an architecturally significant lever, and a RAG system has more than one place where caching can meaningfully help, each with a different risk profile worth understanding before adding it. Embedding caching — storing the vector for a query that’s been seen before, rather than recomputing it — is low-risk and often high-value, since embedding computation is a cost that repeats identically for repeated or near-identical queries. Retrieval result caching — storing the actual chunks returned for a query — is somewhat higher-risk, because it can serve stale results if the underlying knowledge base has changed since the cached result was stored, which means retrieval caching needs an invalidation strategy tied to the same content-update events that trigger re-ingestion, not an independent, disconnected expiration policy that might leave a cached result valid well past the point where the underlying source content it was built from has already changed.

Caching full generated responses is the highest-risk form, appropriate only for repeated, identical queries against unchanged underlying context, since caching a full response risks serving an answer that’s gone stale in a way that’s much harder for a user to detect than a slightly-off retrieval result would be — a cached final answer looks exactly as confident and complete as a freshly generated one, with nothing to signal that it might not reflect the current state of the underlying knowledge base. Architecting caching thoughtfully, with invalidation tied explicitly to the same underlying data-change events discussed in ingestion, is what separates caching that meaningfully improves latency and cost from caching that quietly reintroduces the exact staleness problem the rest of the architecture was designed to avoid.

Scaling each component according to its load pattern

A modular architecture’s payoff shows up specifically in how it scales, because each component in a RAG system has a different load profile, and an architecture that lets each one scale independently handles growth considerably more gracefully than one that scales everything together as a single unit. The vector store’s scaling concerns are largely about data volume and query throughput — sharding or replicating as the number of indexed chunks and the query rate both grow. The retrieval service scales primarily with query volume, and because it’s usually stateless — not holding onto data between requests — it’s typically straightforward to scale horizontally by simply running more instances behind a load balancer.

The generation service’s scaling story is different again, and often the least within an architecture’s direct control, since it usually means calls to an external language model provider, whose rate limits and availability the architecture has to design around — with request queuing, retry logic, and potentially multiple provider fallbacks — rather than something the architecture can scale purely by adding more of its infrastructure. And the ingestion pipeline, as discussed earlier, scales along a completely different axis: burst capacity for large document updates, rather than the smooth, continuous throughput scaling that characterizes the query-serving side of the system. Recognizing that these four components need different scaling strategies, rather than treating “scaling the RAG system” as one undifferentiated problem, is what lets an architecture grow into production load without every component becoming a bottleneck at the same time for entirely different underlying reasons.

Where authentication, authorization, and secrets fit in the architecture

Every component discussed so far handles some part of turning a question into an answer, but a production RAG architecture also has to answer a question that has nothing to do with retrieval or generation quality at all: who is allowed to ask what, and who is allowed to see the results. This needs to be architected as its own concern, threaded consistently through every component, rather than left to whichever single component happens to be closest to the user-facing edge of the system.

Authentication — confirming who a request is coming from — typically belongs at the entry point to the orchestration layer, before any retrieval or generation work happens at all, so that an unauthenticated request never reaches the more expensive downstream components in the first place. Authorization — deciding what that authenticated user or system is allowed to access — is architecturally trickier, because it needs to be enforced not just at the orchestration layer’s entry point but all the way down at the retrieval layer, as a pre-filter on what the vector store is allowed to return, for exactly the reason discussed elsewhere: a permission check applied only after retrieval has already happened is a meaningfully weaker guarantee than one enforced as part of the retrieval query itself. This means the orchestration layer needs to reliably pass a user’s permission scope down to the retrieval service on every single request, and the retrieval service needs to apply that scope as a hard filter rather than as an optional parameter that a caller could theoretically omit.

Secrets management — API keys for the generation service, credentials for the vector store, any keys needed by ingestion connectors to reach source systems — deserves its architectural attention too, and the practical guidance here is unglamorous but important: secrets belong in a dedicated secrets-management system, injected into each component at runtime, never hardcoded into application code or checked into version control alongside it. This matters more in a RAG architecture with several independently deployed components than it does in a simple monolith, since each additional service is another place secrets need to be correctly provisioned, correctly rotated, and correctly kept out of logs — logs which, given how much of this architecture’s value comes from detailed tracing, are exactly the kind of place a secret can end up leaking into if request and response logging isn’t deliberately designed to scrub sensitive fields before anything gets written to a persistent log store.

Multi-tenant architecture: isolating tenants across every layer

For a RAG system serving multiple customers, business units, or user groups from shared infrastructure, multi-tenancy needs to be an architectural decision made consistently across every component discussed so far, not a single filter added at one convenient layer and assumed to cover the whole system. At the vector store, this typically means tenant-isolated partitions or collections, so that a similarity search cannot return another tenant’s data, structurally, rather than relying entirely on an application-level filter applied correctly on every single query across every code path that ever touches the vector store.

At the retrieval service, tenant scope needs to be a required parameter on every request, enforced as part of the query rather than layered on afterward, and at the ingestion pipeline, tenant identity needs to be tracked and attached to every piece of content from the moment it’s ingested, so there’s never a window where content exists in the system without a clear, correctly-assigned tenant scope attached to it. Architecting multi-tenancy this way — as a property enforced structurally at each layer, rather than as a single filter bolted onto one layer and trusted to protect everything above and below it — is what prevents the high-consequence failure mode of one tenant’s data surfacing in another tenant’s results, a failure that’s particularly damaging in this kind of system precisely because a wrong but plausible-sounding answer built from another tenant’s data gives no obvious visual signal to a user that anything crossed a boundary it shouldn’t have.

Deployment topology: managed services versus self-hosted components

Every component discussed so far can be either a managed service a team consumes as an API, or infrastructure the team runs and operates themselves, and this deployment decision is worth making deliberately for each component rather than defaulting uniformly to one approach across the whole architecture. A managed vector database removes the operational burden of running and scaling that infrastructure, at the cost of less control over its internals and an ongoing dependency on a third party’s availability and pricing. A self-hosted vector database offers more control and can be more cost-effective at large scale, at the cost of a team having to build and maintain operational expertise in running it well.

The generation service is almost always consumed as a managed API against a third-party model provider for most teams, simply because self-hosting a competitive large language model is a substantial undertaking most organizations have no reason to take on themselves, unless data residency, cost at extreme scale, or model customization needs make self-hosting necessary rather than merely theoretically possible. A pragmatic architecture often lands on a mixed topology — a managed generation service, because self-hosting a competitive model rarely makes sense, paired with either a managed or self-hosted vector store depending on the team’s operational capacity and how much control over that piece of infrastructure matters for their use case — rather than treating “managed” or “self-hosted” as an all-or-nothing choice that has to apply identically across every single component in the architecture.

Building observability into the architecture from the start

Given how many components a request passes through, and how many of the mistakes discussed throughout this page — a stale cache, a permission gap, a mismatched embedding model — are the kind that produce no traditional error at all, observability needs to be architected as a cross-cutting concern from the beginning rather than added once an incident makes the gap in visibility obvious. This means every component discussed so far needs to emit structured, correlated logs and traces tagged with a shared request identifier, so that a single user request’s full journey — through query rewriting, retrieval, reranking, prompt assembly, and generation — can be reconstructed as one coherent trace rather than as disconnected fragments sitting in separate logging systems for each separate component.

Architecturally, this usually means standardizing on a common tracing format and a shared observability backend across every component, even when those components are built with different technologies or run by different teams, because the value of end-to-end tracing collapses the moment even one component in the chain logs in an incompatible format that can’t be correlated with the rest. It also means deciding early which data gets captured at each stage — the actual retrieved chunks, the assembled prompt, the raw model output before any post-processing — since retrofitting this level of detail onto an already-running system is considerably harder than building it in as each component is first developed, when adding a structured log line costs almost nothing compared to the cost of trying to reconstruct that same visibility after the fact, once a production problem has already made the absence of it painfully obvious.

Versioning and safely evolving the architecture over time

A RAG architecture, once live, rarely stays static — embedding models improve and get swapped out, chunking strategies get refined, new reranking approaches get introduced — and a well-designed architecture needs a deliberate way to evolve each of these pieces without breaking what’s already in production, which raises versioning as its architectural concern distinct from any single component’s internal design. Changing an embedding model is the case that most needs explicit handling, because a new model’s vectors aren’t compatible with an old model’s — comparing a query embedded by a new model against a corpus embedded by an old one produces meaningless similarity scores, even though nothing about the request looks obviously broken.

Handling this well typically means maintaining the ability to run two versions of an embedding pipeline side by side during a migration: re-embedding the existing corpus with the new model in the background while the old version continues serving live traffic, then cutting over to the new version only once the new embeddings are fully in place and validated, rather than switching over mid-migration and serving a mix of old and new vectors that were never designed to be compared against each other in the first place. The same principle extends to chunking strategy changes, prompt template changes, and reranking model changes — each of these is, in effect, a data or logic migration that needs the same deliberate, staged rollout discipline a team would apply to any other significant production data migration, rather than a configuration tweak that gets deployed instantly across the whole system with no transition period and no way to verify the new version is behaving as expected before it’s fully live.

A worked example: architecture for a documentation search assistant

These decisions are easier to see clearly against one concrete case, so consider the architecture for a documentation search assistant serving a moderate but real amount of production traffic. Ingestion runs as an asynchronous, queue-based pipeline, triggered whenever documentation source files change, chunking and embedding new or updated content and writing it into a managed vector store — chosen managed specifically because the team doesn’t want to take on vector infrastructure operations themselves for a system at this particular scale. The retrieval service is its own small, independently deployable service, combining hybrid search and a lightweight reranking step, with a stable API the orchestration layer calls without needing to know any of retrieval’s internal implementation details.

The orchestration layer, running as a right-sized, horizontally scaled set of stateless instances behind a load balancer, handles prompt assembly with an explicit, tested fallback path for when retrieval returns nothing usable, and calls out to a managed language model API for generation, with retry logic and a secondary provider configured as a fallback for the rare case where the primary provider is unavailable. Embedding and retrieval-result caching are both in place, with invalidation tied directly to the same content-change events that trigger re-ingestion, while full-response caching is deliberately left out entirely, since this particular assistant’s query patterns are varied enough that the caching hit rate wouldn’t justify the staleness risk it would introduce — a judgment call worth revisiting periodically as usage patterns become clearer, rather than a decision made once during initial design and never reconsidered again as the system matures. Every component logs its stage-level traces into a shared observability pipeline, so a slow or wrong answer can be traced back to whichever stage caused it, rather than showing up only as an undifferentiated “the assistant gave a bad answer” report with nothing underneath it to investigate. Authorization is enforced at the retrieval service specifically, scoping every search to the roles and teams the requesting user belongs to, so that a question from one department’s employee cannot surface another department’s restricted documentation, regardless of how conceptually similar the two departments’ content happens to be. And when the underlying embedding model eventually gets upgraded — as it inevitably will, once a meaningfully better model becomes available — the team runs the old and new embedding pipelines side by side during a deliberate migration window, validating that retrieval quality on the new embeddings matches or exceeds the old ones before cutting query traffic over, rather than switching instantly and risking a period where queries and the corpus are compared using incompatible vector spaces.

Common architectural mistakes

A consistent set of mistakes shows up across RAG architectures once they move from a demo into production use, worth naming directly because each one traces back to an architectural decision that seemed reasonable in isolation but didn’t hold up under load or iteration pressure. The most common is running ingestion synchronously within the same request path and infrastructure that serves live queries, which means a large ingestion burst can degrade response times for users actively querying the system at the exact same moment, simply because the two workloads were never architecturally separated in the first place.

A second common mistake is scattering retrieval logic across multiple layers of the system rather than isolating it behind a stable, well-defined retrieval service, which turns every retrieval-quality improvement into a change that has to touch code in several different, loosely related places rather than one well-scoped one. A third is caching full generated responses without an invalidation strategy tied to actual content changes, which quietly reintroduces staleness in the highest-visibility part of the system — the final answer a user sees — while looking, from a monitoring dashboard’s perspective, exactly like a healthy cache hit. A fourth is treating every component’s scaling needs as identical, provisioning ingestion, retrieval, orchestration, and generation as if they all faced the same load pattern, which leads to either wasted capacity on components that never needed it or, worse, an under-provisioned component quietly becoming the bottleneck precisely because nobody separated out its different scaling profile from everything else. And a fifth, tying back to the very first architectural decision covered here, is jumping straight to a fully modular, microservices-based architecture before any single component has demonstrated a concrete need for that independence — paying coordination overhead from day one for flexibility the system doesn’t yet, and may never need.

What connects every one of these mistakes, and every architectural decision discussed before them, is a consistent theme: a RAG system’s actual reliability, performance, and maintainability are determined far more by how deliberately its components are separated, scaled, secured, and evolved over time than by any single choice of vector database or language model. Two teams using identical underlying technology can end up with meaningfully different systems in production, purely based on whether ingestion was decoupled from query-serving, whether retrieval logic was cleanly isolated behind a stable interface, whether multi-tenancy was enforced structurally at every layer rather than bolted onto one, and whether observability was built in from the start rather than retrofitted after an incident made its absence obvious. None of these decisions are exotic or require unusual expertise to get right — they’re the same kind of deliberate systems-design discipline that makes any complex, multi-component system reliable — but applying that discipline specifically to the components and failure modes unique to retrieval-augmented generation is what distinguishes a RAG architecture built to last from one that merely happened to work well enough for its first demo.