How do you design RAG systems?

Summary slides

Why RAG design starts with the corpus, not the model

Teams new to retrieval augmented generation tend to start by picking a vector database and an embedding model, then treat the corpus itself as a fixed input that simply gets fed through whatever pipeline they’ve assembled. This ordering produces systems that work acceptably on the documents the team happened to test with and then degrade unpredictably the moment production content arrives, because the corpus’s actual structure, and not the choice of vector database, is what determines nearly every downstream design decision that follows it.

A corpus made up of long, narrative documentation reads completely differently than a corpus made up of short, structured product listings, which reads differently again from a corpus of legal contracts with deeply nested clause references, or a corpus of customer support transcripts full of informal, fragmentary language. Each of these calls for a different chunking granularity, a different metadata schema, and often a different retrieval strategy altogether, so the first design decision in any RAG system is not which vector database to use but a close, honest look at what the actual source material looks like, how it’s structured, how much internal cross-referencing it contains, and how a person would naturally look something up in it if they were searching by hand.

This groundwork also surfaces constraints that shape everything after it: whether the corpus updates frequently enough to need incremental re-indexing rather than periodic full rebuilds, whether different documents carry different levels of trust or recency that retrieval should weight accordingly, and whether the corpus contains near-duplicate content that will need deliberate deduplication rather than being allowed to flood retrieval results with several near-identical passages that all say roughly the same thing.

How chunking strategy determines what retrieval can ever find

Once the corpus itself is well understood, chunking is the decision that most directly bounds what retrieval can possibly succeed at, because a retriever can only return whole chunks, never fragments smaller than a chunk or spans larger than what a single chunk contains, so any information split awkwardly across a chunk boundary becomes effectively invisible to the system no matter how good the embedding model or the ranking logic downstream turns out to be.

Fixed-size chunking, splitting a document into equal token windows regardless of its actual structure, is simple to implement and works reasonably well on uniform prose, but it routinely cuts through the middle of a table, splits a step in a numbered procedure from the steps before and after it, or separates a claim from the caveat that qualifies it, each of which quietly damages retrieval quality in ways that are hard to detect until a user reports a wrong or incomplete answer. Structure-aware chunking, splitting along a document’s actual headings, list boundaries, and paragraph groupings instead of a fixed token count, produces chunks that are more semantically coherent and considerably more likely to contain a complete, self-sufficient answer to a plausible question, at the cost of more chunking-pipeline complexity and less predictable chunk sizes that downstream context-budget planning has to accommodate.

Overlap between adjacent chunks, carrying a small amount of the preceding chunk’s text into the next one, helps recover some of what fixed-size chunking loses at boundaries, though it also means near-duplicate text appears across multiple chunks in the index, which the retrieval and deduplication logic needs to account for explicitly rather than simply returning both overlapping chunks as if they contained independent information. Many production systems combine the two approaches: structure-aware splitting as the primary boundary, with a fallback to size-based splitting whenever a single structural section is too long to fit comfortably inside one chunk on its own, giving retrieval both semantic coherence where the document structure supports it and a predictable upper bound on chunk size everywhere else.

How embedding model choice shapes retrieval quality more than most teams expect

Every chunk needs to become a vector before retrieval can compare it against an incoming query, and the embedding model doing that conversion has an outsized effect on retrieval quality precisely because its errors are silent: a poorly matched embedding model doesn’t throw an error, it simply returns chunks that are less relevant than they should be, ranked in a plausible-looking order, so the system appears to be working right up until someone checks whether the returned material was the best available match.

General-purpose embedding models trained broadly across the open web perform reasonably on generic natural-language queries but often underperform noticeably on domain-corpora full of specialized terminology, internal product names, or industry jargon that the model rarely or never encountered during its training, because the model’s notion of semantic similarity was shaped by a distribution of text that doesn’t closely resemble the corpus it’s now being asked to represent. Domain-adapted or fine-tuned embedding models close this gap meaningfully, at the cost of a training and maintenance investment that only pays off once retrieval quality on the general-purpose model has been measured and found wanting, rather than assumed to be a problem in advance.

Embedding dimensionality is a related, often overlooked lever: higher-dimensional embeddings can encode finer semantic distinctions but cost more to store and search over at scale, while lower-dimensional embeddings are cheaper and faster but collapse some distinct concepts into vectors that sit closer together than they should. The right dimensionality for a system depends on corpus size and query volume more than on any universal rule, which is exactly why this decision, like chunking, benefits from being validated against the system’s actual retrieval quality metrics rather than carried over unexamined from whatever configuration a tutorial or a competitor happened to use.

How hybrid retrieval catches what pure vector search misses

Pure semantic vector search is strong at matching conceptually related text even when the wording differs completely, but it is often surprisingly weak at exact-match retrieval: a query containing a product code, an error message, an acronym, or a precise numeric value can retrieve semantically similar but factually wrong chunks, because the embedding space represents meaning rather than literal string content, and a product code that never appeared in the embedding model’s training data has no reliable semantic representation to begin with.

Keyword-based retrieval, using an inverted index and a ranking function like BM25, handles exactly this case well, because it matches on literal terms rather than learned semantic similarity, at the cost of missing related content that happens to use different wording than the query. Hybrid retrieval, running both a vector search and a keyword search in parallel and then combining their results through a fusion step such as reciprocal rank fusion, captures the strengths of each approach: the keyword search reliably surfaces exact-match content that the vector search would have missed, while the vector search reliably surfaces conceptually related content that the keyword search’s literal matching would have missed.

The combination step itself deserves design attention rather than being treated as a mechanical afterthought once both result sets exist, because a naive combination, such as simply interleaving the two ranked lists, can end up burying a highly relevant keyword match beneath several mediocre vector matches, or the reverse. A weighted fusion that accounts for how confident each retrieval method was in its top results, rather than treating every position in every ranked list as equally meaningful, tends to produce noticeably better final rankings than either a naive interleave or either retrieval method used entirely on its own.

How reranking refines a retriever’s first pass into something the model can trust

The initial retrieval pass, whether vector, keyword, or hybrid, typically optimizes for recall across a fairly large candidate set, deliberately casting a slightly wider net than what will be sent to the model, because retrieval at that stage is cheap per candidate and missing a relevant chunk entirely is a much worse failure than including a few mediocre ones. Reranking is the step that then narrows that wider candidate set down to the smaller, more precise set that gets assembled into context, using a model specifically trained to judge fine-grained relevance between a query and a single candidate passage far more accurately than the coarser similarity score the initial retrieval pass produced.

Cross-encoder rerankers, which process the query and each candidate passage together rather than comparing independently computed embeddings, are considerably more accurate at this fine-grained judgment than the bi-encoder models typically used for the initial retrieval pass, precisely because they can attend directly to the interaction between the query and the passage rather than relying on two separately computed vectors that were never compared to each other during encoding. This accuracy comes at a computational cost, since a cross-encoder has to run inference on every query-candidate pair individually rather than doing one fast vector comparison, which is exactly why reranking is applied to a smaller narrowed-down candidate set rather than the full corpus, letting the cheaper initial retrieval pass do the broad filtering and the more expensive reranker do the fine discrimination on a manageable shortlist.

Skipping reranking entirely is a viable choice for smaller corpora or latency-sensitive applications where the initial retrieval pass’s ranking is already good enough, but for larger or more ambiguous corpora, where many chunks share surface-level similarity to a query without all of them being equally useful, reranking is frequently the single highest-leverage addition a team can make to an underperforming RAG system, often improving perceived answer quality more than switching to a larger or more expensive generation model would.

How context assembly turns retrieved chunks into something a model can use well

Retrieval and reranking together decide which chunks make it into the model’s context, but a separate, equally consequential decision remains: how those chunks are assembled into the prompt the model receives, including their ordering, their formatting, how source attribution is preserved so the model can cite what it used, and how much of the available context budget goes to retrieved material versus the system instructions and the user’s query.

Ordering matters because models exhibit measurable positional effects across long context, tending to weight information near the beginning and end of a long input more heavily than information buried in the middle, so simply concatenating chunks in whatever order the reranker returned them, without any further thought, can bury the single most relevant passage in a position the model is statistically less likely to draw on heavily. Placing the highest-ranked chunks near the beginning of the assembled context, and reserving the middle for lower-confidence supporting material, tends to produce noticeably more grounded answers than an arbitrary or purely rank-descending order that ignores this positional effect entirely.

Preserving clear source boundaries and attribution within the assembled context, rather than concatenating chunk text into one undifferentiated block, lets the model reason explicitly about which claim came from which source and lets it generate accurate citations back to the original material, which matters enormously for any application where a user needs to verify an answer against its underlying source rather than simply trusting the model’s synthesis at face value. This formatting decision also directly supports honest uncertainty: a model that can see clearly that only one thin, tangentially related source was retrieved is far better positioned to hedge appropriately than a model handed an undifferentiated wall of text that gives no signal about how much support exists behind the answer it’s about to produce.

How grounding and citation requirements change what the generation step must do

The generation step in a RAG system isn’t simply producing a fluent answer, it’s producing an answer that stays faithful to what was retrieved, which is a meaningfully different and considerably harder constraint than open-ended generation, because a fluent, plausible-sounding answer that draws on the model’s general training rather than the retrieved material defeats the entire purpose of retrieval augmentation even when nothing about the output looks obviously wrong to a reader who can’t check it against the source.

Explicit grounding instructions in the system prompt, directing the model to answer only from the provided context and to say plainly when the provided context doesn’t contain enough information to answer confidently, measurably reduce this kind of ungrounded generation, though they don’t eliminate it entirely on their own, because a sufficiently capable model can still blend its general knowledge with the retrieved material in ways that are difficult to detect from the output text alone. Citation requirements, asking the model to attribute claims to retrieved sources inline, provide an additional, more checkable layer of grounding, both because generating a citation is harder for a model to do convincingly when it’s drawing on unsupported general knowledge, and because the citations themselves give a downstream reader or an automated evaluation process something concrete to verify the answer against.

The tension in this stage is that stronger grounding constraints generally come at some cost to naturalness and occasionally to useful synthesis, since a model heavily constrained to only restate what’s explicitly present in retrieved passages may produce a more stilted or repetitive answer than one more latitude to synthesize and connect ideas across sources in its words. Finding the right point on that tradeoff for an application, rather than defaulting to either extreme, is itself part of RAG system design, and it typically depends heavily on how much the application’s users need to trust and verify individual claims versus how much they simply want a fluent, useful synthesis they don’t intend to check line by line.

How to handle queries that retrieval can’t answer well

Every RAG system eventually receives queries the corpus simply doesn’t have good material for, whether because the question falls entirely outside the corpus’s actual coverage, because the relevant information exists but is phrased in the corpus in a way that doesn’t surface well under the system’s current retrieval strategy, or because the question is ambiguous enough that no single retrieval pass could reliably guess which of several plausible interpretations the user meant.

The naive failure mode here is a system that always assembles some context, however weak, and always generates a confident-sounding answer regardless of how thin the underlying support was, because nothing in the pipeline explicitly checks whether what was retrieved is good enough to answer from. A more deliberate design introduces an explicit confidence or relevance threshold on the retrieval and reranking scores, below which the system either asks a clarifying question, states plainly that it doesn’t have enough information in the available material, or falls back to a broader, less precise search rather than confidently answering from clearly insufficient context, treating retrieval failure as a distinct, plannable state rather than something the generation step has to quietly paper over on its own.

Query ambiguity deserves separate handling from outright retrieval failure, because an ambiguous query often does retrieve confidently relevant material, just material relevant to the wrong one of several plausible interpretations, which a pure relevance-score threshold won’t catch on its own. Detecting this case usually requires either an explicit disambiguation step that asks the model to identify multiple plausible interpretations of the query before retrieval runs, or a broader initial retrieval pass that deliberately samples across the different interpretations so the system can notice, before generation, that the top results cluster into different topics rather than one coherent one.

How to evaluate a RAG system without conflating retrieval quality and generation quality

Evaluating only the final answer a RAG system produces, without separately evaluating what was retrieved, makes it impossible to tell whether a disappointing answer traces back to weak retrieval that never surfaced the right material in the first place, or to weak generation that had good material available and still failed to use it well, and those two failure modes call for completely different fixes, so conflating them in evaluation routinely sends teams chasing the wrong improvement.

Retrieval-evaluation typically measures whether the chunks known to contain the answer to a test query appear within the retrieved set, and at what rank, using metrics like recall at a cutoff and mean reciprocal rank against a curated set of query-answer pairs with known correct source chunks. This evaluation can and should run independently of the generation step entirely, since it only requires checking retrieval’s output against a known-correct set of source chunks, which makes it considerably cheaper and faster to iterate on than end-to-end evaluation that requires judging generated text quality.

Generation-evaluation, retrieval held constant, typically checks faithfulness, whether the generated answer’s claims are supported by the retrieved context it was given, and completeness, whether the answer uses the relevant material that was available to it rather than ignoring good context in favor of a thinner, less complete response. Running both evaluations separately, rather than only measuring end-to-end answer quality against a reference answer, lets a team isolate a detected quality regression to the stage responsible and fix that stage directly, instead of guessing at which of several pipeline stages introduced the regression and iterating on the wrong one.

How corpus freshness and update strategy shape long-term system design

A RAG system built against a static, unchanging corpus can treat indexing as a one-time batch job, but very few corpora stay static for long: documentation gets revised, product catalogs change, policies get updated, and a system that never re-indexes will keep confidently retrieving and citing content that’s since become outdated or outright wrong, which is often a worse failure mode than admitting uncertainty, because a confidently cited stale answer looks exactly as trustworthy as a confidently cited current one to a reader who can’t tell the difference.

Full periodic re-indexing, rebuilding the entire vector index from the current corpus on some regular schedule, is simple to reason about and guarantees the index never drifts too far from the source material, but it scales poorly as a corpus grows large, since re-embedding and re-indexing the entire corpus repeatedly becomes an increasingly expensive and slow operation that eventually can’t keep pace with how frequently the underlying content changes. Incremental indexing, detecting and re-embedding only the documents or chunks that changed since the last update, scales considerably better but requires infrastructure investment: reliable change detection against the source content, careful handling of chunk boundaries that shift when a document is edited, and a way to remove or supersede stale chunks from the index rather than letting old and new versions of the same content both remain retrievable simultaneously.

Recency itself is also a retrieval signal worth designing for explicitly rather than leaving implicit, particularly in corpora where older content isn’t simply superseded but becomes less relevant over time, such as a support-ticket archive or a changelog; incorporating a recency-aware ranking adjustment, rather than relying purely on semantic similarity to a query that says nothing about how current the answer needs to be, helps ensure the system surfaces the currently accurate version of an answer rather than an equally similar but outdated one sitting elsewhere in the same corpus.

How agentic and multi-step retrieval extend RAG beyond a single retrieval pass

A single retrieval pass, however well tuned, assumes the information needed to answer a query can be found through one query-shaped search against the corpus, which holds for a large share of queries but breaks down for questions that require synthesizing information found through several successive, dependent searches, where the second search can’t even be formulated until something learned from the first search result is known.

Agentic retrieval addresses this by letting the model itself decide, based on what an initial retrieval pass returned, whether it has enough information to answer or whether it needs to issue a further, differently phrased search to fill a gap it can identify in what it currently has, repeating this process for as many rounds as the query requires rather than being limited to exactly one retrieval pass regardless of how complex the underlying question turns out to be. This considerably expands what a RAG system can handle, letting it answer multi-hop questions that require connecting facts found in separate parts of the corpus, at the cost of added latency, added unpredictability in how many retrieval rounds a query will need, and a new failure mode where the model issues redundant or poorly targeted follow-up searches that don’t close the gap it identified.

Designing for this pattern well means giving the model clear guidance on when to stop searching and answer with what it has, rather than searching indefinitely, and giving it visibility into what it has already retrieved across all prior rounds so a later search doesn’t simply re-retrieve the same material under slightly different phrasing. It also means the context-budget planning discussed earlier has to account for material accumulated across multiple retrieval rounds rather than assuming a single, fixed-size retrieval pass, since an agentic retrieval loop that runs for several rounds can otherwise consume far more of the available context budget than a single-pass system ever would.

How cost and latency constraints shape which of these choices are viable

Every design choice discussed so far, hybrid retrieval, cross-encoder reranking, agentic multi-step retrieval, has a cost and latency profile, and very few production RAG systems can afford to adopt every available technique at its most thorough setting, which means cost and latency constraints are not an afterthought bolted onto a finished design but an input into which techniques are worth adopting for an application in the first place.

An interactive, user-facing application where a person is actively waiting for a response has a much tighter latency budget than a background batch process generating reports overnight, which directly changes what’s viable: a full cross-encoder reranking pass over a large candidate set, or an agentic retrieval loop that might run several sequential rounds, can be entirely reasonable in the batch case and unacceptable in the interactive one, where every added retrieval round or reranking pass shows up directly as time the user spends staring at a loading indicator.

Cost scales similarly with retrieval volume and reranking thoroughness, and the temptation to simply retrieve more candidates and rerank more thoroughly on the assumption that more retrieved material can only help quality runs directly into this constraint, because beyond a certain point additional retrieved material stops meaningfully improving answer quality while continuing to add measurable cost, which is exactly why the evaluation practices discussed earlier matter here too: measuring where retrieval quality plateaus for a corpus and query distribution lets a team set retrieval and reranking parameters at the point that captures nearly all of the achievable quality without continuing to pay for diminishing returns beyond it.

How access control and multi-tenancy complicate retrieval beyond pure relevance

Everything discussed so far assumes retrieval only has to decide what’s relevant, but in many deployments it also has to decide what a user is permitted to see, and treating that as a filter applied after relevance ranking, rather than as a constraint on retrieval itself, opens a serious failure mode: a highly relevant but access-restricted chunk can rank near the top of the candidate set, get stripped out by a late permission check, and quietly leave the user with a noticeably thinner, less complete answer than someone with broader access would have received for the identical query, with no signal anywhere in the response that anything was withheld.

The more robust approach filters by permission before or during retrieval itself, not after, typically by attaching access metadata directly to each indexed chunk and incorporating that metadata into the retrieval query so that a user’s search only ever ranks against the subset of the corpus they’re authorized to see in the first place, rather than ranking against the full corpus and hoping a downstream filter catches everything it needs to. This matters even more once reranking and generation are involved, because a cross-encoder reranker or a generation model that never sees a restricted chunk can’t accidentally leak its content through a paraphrase or an inferred detail, whereas a chunk that reaches those later stages and only gets blocked at the very last step carries a risk of its content surfacing indirectly in the model’s synthesis even if the original chunk text itself is never shown.

Multi-tenant corpora, where multiple organizations or customers share the same underlying RAG infrastructure but must never see each other’s content, raise the stakes on this further, since a failure here isn’t just a degraded answer but a data isolation breach, and it’s exactly why access-aware retrieval needs to be designed as a first-class part of the indexing and retrieval architecture from the outset, validated with the same seriousness as any other security boundary in the system, rather than treated as a filtering detail that can be layered on after the rest of the pipeline is already working.

Common mistakes teams make when designing RAG systems

A first mistake, and one of the most common, is choosing chunk size and overlap once during initial setup based on a rule of thumb or a default from a tutorial, and never revisiting that choice against the system’s actual corpus and actual retrieval quality metrics, leaving a poorly matched chunking strategy quietly capping retrieval quality indefinitely.

A second mistake is relying on vector search alone without any keyword component, missing exact-match queries involving codes, identifiers, or terminology that a purely semantic embedding space has no reliable way to represent.

A third mistake is skipping reranking entirely on the assumption that a good enough embedding model makes it unnecessary, when in practice reranking is often the single highest-leverage addition available to an underperforming system, particularly on larger or more ambiguous corpora.

A fourth mistake is concatenating retrieved chunks into the model’s context in whatever order the retriever happened to return them, ignoring the positional effects long-context models exhibit and burying the most relevant material in a position the model is statistically less likely to weight heavily.

A fifth mistake is stripping source attribution out of the assembled context for the sake of a cleaner prompt, losing the model’s ability to generate accurate citations and losing an important signal the model could otherwise use to reason about how much support exists behind a claim.

A sixth mistake, closely related, is never giving the model explicit grounding instructions or citation requirements at all, leaving it to blend retrieved material with its general training in ways that are difficult to detect and that defeat much of the actual purpose of retrieval augmentation.

A seventh mistake is building a system that always confidently answers regardless of retrieval quality, with no explicit relevance threshold or fallback behavior for the case where nothing relevant was retrieved.

An eighth mistake is treating query ambiguity as if it were the same problem as retrieval failure, when an ambiguous query often retrieves confidently relevant material for the wrong interpretation entirely, a case a simple relevance threshold won’t catch.

A ninth mistake is evaluating only end-to-end answer quality without separately evaluating retrieval, making it impossible to tell whether a detected quality problem traces back to weak retrieval or weak generation, and leaving a team unable to fix the actual responsible stage directly.

A tenth mistake is treating the corpus as a static, one-time indexing job in a domain where the underlying content changes, letting the index drift silently out of date and letting the system keep confidently citing material that’s since become outdated or wrong.

An eleventh mistake is ignoring recency as an explicit retrieval signal in a corpus where older content becomes less relevant over time, relying purely on semantic similarity that says nothing about how current a matched passage is.

A twelfth mistake is adopting agentic multi-step retrieval without any explicit stopping condition, letting the model search indefinitely or issue redundant follow-up searches that never close the gap it originally identified.

A thirteenth mistake is adding every available technique, hybrid retrieval, thorough reranking, multi-round agentic search, at maximum thoroughness without weighing the cost and latency each one adds, rather than measuring where retrieval quality plateaus for the system’s corpus and setting parameters at that point instead of continuing to pay for returns that have already diminished to nothing.

A fourteenth and final mistake is treating access control as a filter applied after relevance ranking rather than as a constraint on retrieval itself, letting restricted content reach reranking or generation before being blocked at the last possible step, which risks its content surfacing indirectly through the model’s synthesis even when the original chunk is never shown directly to the user.

What connects all fourteen of these mistakes is treating one stage of the RAG pipeline as if it could be designed and tuned in isolation from the others, when in practice chunking bounds what retrieval can find, retrieval bounds what reranking can refine, context assembly bounds what generation can faithfully use, and evaluation only becomes actionable once it can distinguish which of these stages a failure traces back to. Teams that design and evaluate the full pipeline together, with explicit thresholds for uncertainty, explicit grounding requirements, and a clear-eyed view of the cost each added technique buys in retrieval quality, tend to build RAG systems that stay reliable and cost-proportionate as both their corpus and their query volume grow, rather than systems whose early promising results on a small test set quietly stop holding once production traffic and an evolving corpus arrive.