What is RAG?

Summary slides

The problem that makes RAG necessary

A large language model learns everything it knows during a training process that ends at some fixed point in time. Once training is finished, the model’s knowledge is frozen — it has no built-in way to learn about anything that happened afterward, no access to a company’s internal documents, no visibility into a user’s private data, and no mechanism for updating a fact it got wrong without being retrained from scratch. Retraining a large model is expensive, slow, and not something you can realistically do every time a document changes or a new fact needs to be added. That gap between “what the model knows” and “what the model needs to know to answer this question well” is the entire reason RAG exists.

The gap shows up in two closely related but distinct failure modes. The first is a knowledge gap: the model simply was never trained on the information needed to answer correctly, whether that’s a company’s product documentation, a legal contract, yesterday’s news, or a customer’s account history. The second is a confidence problem: language models are trained to produce fluent, plausible-sounding text, and when they don’t know an answer, that training pressure doesn’t disappear — it just produces a fluent, plausible-sounding wrong answer instead of an honest “I don’t know.” This is what gets called hallucination, and it’s particularly dangerous precisely because a hallucinated answer often reads exactly as confidently as a correct one. A model with no access to your refund policy won’t necessarily say “I don’t have that information” — it might just generate something that sounds like a refund policy, built out of patterns it learned from other companies’ policies during training.

Both of these problems point toward the same fix: give the model the actual, current, relevant text at the moment it’s asked a question, rather than asking it to reconstruct that information from a frozen and necessarily incomplete memory. That’s the whole idea behind retrieval-augmented generation — not making the model smarter in some abstract sense, but making sure it has the right material in front of it before it has to answer.

How the pipeline works, end to end

Once you accept that the model needs the right material in front of it, the next question is mechanical: how do you find “the right material” out of potentially millions of documents, fast enough to answer a question in a few seconds? This is where RAG’s two-stage structure comes from — a retrieval stage that finds relevant content, followed by a generation stage that turns that content into an answer.

The retrieval stage begins long before any question is ever asked, with a preparation step usually called ingestion. Source documents — PDFs, web pages, support tickets, internal wikis, whatever the knowledge base consists of — get broken into smaller pieces, typically called chunks, because feeding an entire 200-page document to a retrieval system as a single unit makes it nearly impossible to find the one relevant paragraph buried inside it. Each chunk is then converted into a vector embedding: a list of numbers, usually a few hundred to a few thousand of them, produced by an embedding model in such a way that chunks with similar meaning end up as numerically similar vectors, even if they don’t share any of the same exact words. Those vectors get stored in a vector database, indexed in a way that makes it possible to quickly find the vectors closest to any new vector you give it.

When a user asks a question, that question gets converted into a vector using the same embedding model, and the vector database is searched for the chunks whose vectors are closest to the question’s vector — closest, in this context, meaning most similar in meaning, not similar in exact wording. This is the step that lets a question phrased as “how do I get my money back” correctly retrieve a document titled “refund policy” even though the two phrases share almost no words in common. The database returns a ranked list of the most relevant chunks, typically somewhere between three and twenty of them depending on the system.

Those retrieved chunks are then inserted into a prompt alongside the user’s original question, and that combined prompt — question plus supporting context — is what gets sent to the language model. The model’s job at this point is narrower and more constrained than answering from open-ended memory: it’s asked to synthesize an answer using the material it was just handed, ideally citing or staying faithful to that material rather than drifting back into whatever it happens to remember from training. The output the user sees is the result of that generation step, but the quality of that output is determined almost entirely by what happened one step earlier — because a language model, however capable, cannot generate an accurate answer to a question its context doesn’t contain the answer to.

Why retrieval quality is the bottleneck

That last point is worth sitting with, because it reshapes where most of the engineering effort in a RAG system needs to go. It’s tempting to think of the language model as the “smart” part of the system and retrieval as a simple lookup step, but in practice the reverse is closer to true. If retrieval pulls back the wrong chunks — irrelevant ones, outdated ones, or ones that are topically close but don’t contain the answer — no amount of generation-stage sophistication can recover from that. The model will do its best with what it’s given, and what it’s given will be wrong. This is why so much of what makes one RAG system meaningfully better than another comes down to decisions made before generation ever happens: how the data was chunked, which embedding model was used, and whether the retrieval step relies on semantic similarity alone or combines it with something more precise.

Semantic similarity search, the kind vector embeddings provide, is powerful at matching meaning across different phrasing — but it has a predictable weakness. Embedding models represent the general sense of a piece of text well, but they’re comparatively bad at treating an exact string — a product code, an error number, a legal clause identifier, a person’s name — as something that has to match precisely. Ask an embedding-only retrieval system to find “error E-4471” and it might return several chunks about errors in general, ranked by how conceptually similar they are to “error,” without ever surfacing the one chunk that happens to contain the literal string “E-4471,” because that string doesn’t carry much semantic weight to an embedding model the way it does to a human reading it. This gap between what vector search is good at and what precise term-matching requires is exactly the kind of failure that shows up constantly in systems and rarely in demos, because demos tend to ask conceptual questions rather than the exact-code lookups users type.

Chunking: preparing data so retrieval can find it

Understanding that retrieval quality depends this heavily on what gets searched naturally raises the next question: how the source data gets prepared in the first place, since a chunk that’s poorly formed can’t be retrieved well no matter how good the embedding model or search algorithm is downstream. Chunking size is the first and most consequential decision. Chunks that are too large — say, an entire multi-page document treated as one unit — tend to dilute the embedding, because the vector ends up representing an average of many different subtopics rather than any one of them clearly, which makes it harder for a question to match strongly against the paragraph that answers it. Chunks that are too small, on the other hand — a single sentence, for instance — often lose the surrounding context needed to make sense of that sentence on its own; a sentence that says “this only applies to accounts opened after that date” is nearly useless in isolation if the reader (or the retrieval system) has no way to know what “that date” refers to.

Most production systems land somewhere in the range of a few hundred tokens per chunk, often with a modest overlap between consecutive chunks so that information sitting near a chunk boundary doesn’t get orphaned on one side or the other. But fixed-size chunking, splitting text purely by token or character count without regard for its actual structure, is a blunt instrument — it will just as happily cut a chunk off in the middle of a sentence or a table row as at a natural break point. This is what motivates semantic chunking, an approach that splits text at points where the meaning shifts — the boundary between two paragraphs discussing different subtopics, for instance — rather than at an arbitrary character count. Semantic chunking produces chunks that are more likely to be coherent, self-contained units of meaning, which in turn makes them easier for an embedding model to represent well and easier for retrieval to match precisely.

Document structure matters here too, in a way that’s easy to overlook. A well-structured document — one with headings, defined sections, tables with headers — carries information about how its content is organized, and a chunking strategy that respects that structure (keeping a table together, keeping a section under its heading) tends to produce chunks that make more sense both to the retrieval system and, later, to the language model reading them as context. Losing that structure during chunking is a quiet, easy way to degrade a RAG system’s quality without ever touching the retrieval algorithm or the model itself.

Hybrid search: catching what semantic similarity misses

Given that chunking alone can’t fix the exact-term-matching weakness described earlier — a perfectly chunked, perfectly structured document still won’t help an embedding-only system find “error E-4471” if the embedding model just doesn’t weight that string heavily — the natural next step is to address the retrieval mechanism itself. This is what hybrid search does: instead of relying on vector similarity alone, it runs a traditional keyword-based search algorithm, most commonly a scoring method called BM25, in parallel with the vector search, and then combines the two ranked result sets into one.

BM25 is a decades-old, well-understood algorithm that scores documents based on how well their exact terms match the exact terms in a query, with adjustments for how common or rare each term is across the whole collection — a rare term like a product code matching exactly counts for a lot more than a common word like “the” matching. Where vector search is strong at conceptual, paraphrased matching and weak at exact terms, BM25 is exactly the reverse: strong at exact terms, blind to paraphrasing and synonyms. Running both and combining their results plays each method to its strength. A well-implemented hybrid search system typically includes a tunable balance parameter, letting a team weight the two methods differently depending on whether their content and queries lean more conceptual or more exact-term-heavy — a legal document search system searching for clause numbers might weight keyword matching more heavily than a general customer-support assistant fielding open-ended questions would.

There’s a second, related capability that matters just as much in practice: metadata filtering, the ability to narrow a search to only chunks matching a structured condition — a document category, a date range, a product line, a user’s permission level — before or during the similarity search itself, rather than after. Done correctly, this is what’s sometimes called pre-filtering: the filter builds a list of eligible candidates first, and the similarity search only ever ranks among those candidates. Done incorrectly — filtering after the fact, on results already ranked by similarity alone — a restrictive filter can silently return too few results, or in an access-control context, can even risk momentarily surfacing content to a search process that a user shouldn’t be able to see at all, even if it’s stripped out before the final response. Getting this ordering right is a quiet but important correctness property of a well-built retrieval system, not just a performance optimization.

Reranking: a second, more careful look

Hybrid search solves a problem, but it introduces a smaller one of its own: combining two different ranking methods — vector similarity scores and BM25 scores, which are computed in entirely different ways and don’t naturally live on the same numeric scale — requires some kind of fusion strategy, and that fusion is necessarily an approximation. It’s fast, because it has to run across potentially millions of candidate chunks in a fraction of a second, but fast and precise are often in tension. This is where reranking comes in as a second, deliberately more expensive pass, applied only to a much smaller set of candidates — typically the top twenty or fifty results that hybrid search already narrowed things down to, rather than the entire collection.

A reranking model, often a specialized cross-encoder architecture rather than the embedding model used for the initial search, looks at the actual question and each candidate chunk together, as a pair, and produces a much more precise relevance judgment than either vector similarity or keyword overlap can on their own — because it’s evaluating the match between this exact question and this exact chunk, rather than comparing two independently-computed vectors. This precision comes at a computational cost, which is exactly why reranking is applied to a short list rather than the whole database: running a cross-encoder over millions of chunks for every query would be far too slow, but running it over the top few dozen candidates that a cheaper method already surfaced is fast enough for production use while meaningfully improving the final ordering that reaches the language model.

What generation does with what it’s given

With a well-retrieved, well-reranked set of chunks in hand, the pipeline reaches the step most people picture when they think of RAG at all: the language model generating an answer. But it’s worth being precise about what’s happening here, because it explains both RAG’s strength and its limits. The retrieved chunks are inserted into the model’s prompt, typically with the original question repeated alongside them and often with explicit instructions to answer using only the provided context, and sometimes to cite which chunk supported which part of the answer. The model then generates text the same way it always does — token by token, predicting what’s most likely to come next — but now that prediction is conditioned on the retrieved text sitting in its context window, not solely on patterns learned during training.

This is why RAG substantially reduces hallucination without eliminating it as a possibility. The model is now working with current material in front of it, which makes it far more likely to produce an answer that’s grounded in that material rather than a plausible-sounding fabrication built from training-time patterns. But the model can still get it wrong in a few ways: it can misread or misinterpret the retrieved context, it can blend the retrieved material with something it remembers from training in a way that isn’t clearly separated, or — critically — if retrieval failed to surface the right chunk in the first place, generation has nothing correct to work with and may fall back on a plausible-sounding guess anyway, sometimes without any obvious signal to the user that it’s guessing rather than answering from the provided material. This is the honest version of the claim that RAG “solves hallucination” — it dramatically improves the odds, by giving the model something true and specific to ground its answer in, but it depends entirely on retrieval having done its job well, and it doesn’t turn the model into something that mechanically refuses to guess.

How RAG compares to fine-tuning

Understanding what RAG does — supplying relevant, current information at the moment of answering — naturally raises a comparison to the other well-known technique for adapting a language model to a domain: fine-tuning, which involves further training the model itself on a curated dataset so that its internal weights shift toward the target domain’s patterns, vocabulary, and style. The two techniques solve different problems, and confusing them leads to architectural mistakes.

Fine-tuning changes what the model inherently knows and how it behaves — it can teach a model a particular tone, a particular output format, a particular way of reasoning through a certain type of problem, or general familiarity with a domain’s terminology and conventions. What it doesn’t do well is keep the model current with facts that change: every time the underlying information changes, fine-tuning requires retraining, which is slow and expensive compared to just updating a document in a knowledge base that a RAG system will pick up on its very next query. Fine-tuning also doesn’t give the model access to information it was never trained on at all, such as a customer’s private account data — the model’s weights simply don’t and can’t encode facts about data that didn’t exist at training time.

RAG solves precisely the problem fine-tuning struggles with: it keeps answers current by design, since the retrieval step always searches whatever is in the knowledge base right now, and it extends easily to private, per-customer, or rapidly changing data without ever touching the model’s weights. What RAG doesn’t do is change the model’s underlying behavior, tone, or reasoning style — it can only supply better raw material for the model to work with, not make the model itself reason differently about that material. In practice, the two techniques are frequently complementary rather than competing: a team might fine-tune a model to reliably follow a particular answer format or tone, then layer RAG on top of that fine-tuned model to keep its factual answers current and grounded in data. Treating them as an either-or choice usually means picking the wrong tool for at least part of the actual problem.

RAG versus simply using a longer context window

A second comparison follows naturally once fine-tuning is out of the way, because it addresses the same underlying goal from yet another angle: modern language models increasingly support very long context windows, some large enough to fit an entire book’s worth of text directly into a single prompt. If a model can simply hold that much text at once, it’s fair to ask whether retrieval is even necessary anymore — why not just paste the entire knowledge base into the prompt every time and skip the retrieval step altogether?

The honest answer is that long context and retrieval solve overlapping but distinct problems, and long context alone runs into limits that retrieval was specifically designed to avoid. The first is cost and latency: processing a context window with hundreds of thousands of tokens in it, on every single query, is measurably slower and more expensive than processing a short, retrieval-narrowed context of a few thousand tokens, and that cost compounds quickly at any scale of usage. The second is a well-documented phenomenon sometimes called “lost in the middle”: models with very long contexts don’t attend to every part of that context equally well, tending to be more reliable at using information near the beginning or end of the context than information buried in the middle, which means simply stuffing more text into the prompt doesn’t guarantee the model will use the piece of information that matters. The third is scale: even the largest context windows have a ceiling, and a large knowledge base — an enterprise’s full documentation, years of support tickets, a large legal archive — will exceed that ceiling long before it exceeds what a well-indexed retrieval system can search through in milliseconds.

What long context does provide is a wider safety margin and, in some architectures, a useful role as a second-stage check — a system might use retrieval to narrow a huge knowledge base down to a manageable, highly relevant set of chunks, then rely on a long context window to hold more of those chunks than a tighter system could, reducing the risk of a single retrieval step missing something important. The two techniques increasingly work together rather than substituting for one another, with retrieval handling the problem of scale and relevance, and longer context providing more room to work with the results retrieval hands back.

How to tell if a RAG system is working

All of this — chunking decisions, hybrid search, reranking, the choice between RAG and fine-tuning or long context — only matters if there’s a reliable way to know whether the resulting system is good, which raises the question of evaluation. This turns out to be harder than evaluating a traditional software system, because there usually isn’t a single, unambiguous “correct” answer to check against, and because RAG’s two-stage structure means failures can originate in either stage, or both, and it’s easy to misdiagnose a generation problem as a retrieval problem or vice versa without measuring each stage separately.

Retrieval quality is typically measured with metrics like precision — of the chunks retrieved, how many were relevant — and recall — of all the relevant chunks that existed in the knowledge base, how many did retrieval find. A system can have excellent precision and poor recall, confidently returning a few relevant chunks while missing several others that would have mattered, or the reverse, returning lots of marginally relevant chunks alongside the right ones, forcing the generation step to sift through more noise. Generation quality, once the right context has been retrieved, is typically judged on faithfulness — did the answer stay true to what the retrieved context said, or did it drift into unsupported claims — and on relevance to the original question, since an answer can be perfectly faithful to the retrieved context while still failing to address what the user asked.

In practice, teams building systems combine a few approaches: a curated set of test questions with known-correct answers and known-relevant source chunks, run against the system periodically to catch regressions before they reach users; automated scoring using a separate, typically more capable language model to judge faithfulness and relevance at scale, since manually grading every response doesn’t scale; and ongoing monitoring of user interactions — thumbs up or down feedback, follow-up questions that suggest the first answer didn’t land, or explicit corrections — as a continuous signal that a static test set can’t fully capture. None of these approaches alone is sufficient; together, they give a reasonably reliable picture of whether a system is doing its job.

Beyond the basic pipeline: agentic and multi-hop RAG

Everything covered so far describes what’s sometimes called naive or vanilla RAG: one retrieval pass, followed by one generation pass, handling one self-contained question at a time. That structure works well for a large share of questions, but it runs into a predictable limit whenever a question can’t be answered by any single retrieved passage on its own — questions that require combining information from multiple, separate parts of a knowledge base, or that require some amount of reasoning about what to search for before the actual search happens.

Consider a question like “which of our enterprise customers who signed up before our pricing change are still on the old plan.” Answering that well might require first retrieving information about when the pricing change happened, then retrieving a list of enterprise customers, then cross-referencing signup dates against that change, then checking current plan status for each — a chain of related lookups, not a single search. This is the kind of question that motivates more advanced RAG patterns. Multi-hop RAG breaks a complex question down into a sequence of simpler sub-questions, retrieves separately for each one, and combines the results before generating a final answer, rather than trying to answer everything from one retrieval pass. Query rewriting and query expansion address a related but distinct problem: a user’s literal phrasing is sometimes a poor match for how the relevant information is written in the knowledge base, so a rewriting step reformulates the question — sometimes into several variations — before retrieval runs, improving the odds that at least one version of the search finds what’s needed.

Agentic RAG goes a step further, wrapping the entire retrieval-and-generation process inside a reasoning loop where an AI agent decides, dynamically, what to search for, whether the results it got back are sufficient, and whether it needs to search again — potentially with a different query, or against a different data source entirely — before it’s ready to generate a final answer. Rather than a fixed, one-shot pipeline, agentic RAG behaves more like a researcher who checks an initial source, realizes it raises a follow-up question, looks that up too, and only writes up a final answer once enough has been gathered. This flexibility comes at the cost of more latency and more complexity to build and debug correctly, which is why it’s typically reserved for complex question types rather than applied universally — most straightforward factual questions are answered perfectly well by the simpler, single-pass pipeline, and the added complexity of an agentic loop is only worth paying for when the question demands it.

What tends to go wrong when teams build this themselves

Given how many independent decisions go into a working RAG system — chunking strategy, embedding model choice, whether to run hybrid search, whether to add a reranking step, how to filter by metadata, how to evaluate the result — it’s worth being specific about where teams lose quality in practice, because the failure modes are consistent enough to be worth naming directly rather than leaving to be discovered the hard way.

The most common mistake is treating chunking as a one-time, set-and-forget decision made early in a project and never revisited. A chunking strategy that worked reasonably well for a knowledge base of product FAQs often falls apart when the same system is later pointed at long legal contracts or dense technical specifications, because the right chunk size and the right respect for document structure differ by content type. Teams that never go back and re-examine chunking as their content mix evolves tend to see retrieval quality quietly degrade over time without an obvious single cause, because the symptom — vague, slightly-off answers — looks identical to several other possible problems.

A second, closely related mistake is evaluating only the generation step and never the retrieval step in isolation. It’s natural to judge a RAG system by reading its final answers and deciding whether they sound right, but that approach can’t distinguish between “the model reasoned poorly over correct context” and “the model reasoned perfectly over context that was wrong or incomplete in the first place” — and those two failures call for completely different fixes. A team that only ever looks at final answers will often spend significant effort tuning prompts and generation parameters to fix a problem that was sitting in the retrieval stage the whole time, because chunking, embedding choice, or filtering was the point of failure.

A third mistake is assuming semantic search alone is sufficient and skipping hybrid search entirely, usually because early testing happens to focus on conceptual, paraphrase-heavy questions where vector search performs well, and the exact-term failure case never comes up until users start typing in the codes, IDs, and precise terminology that actual usage is full of. This is a case where a system can look good in a demo and then underperform meaningfully once messier queries start arriving — the gap between demo questions and real questions is one of the most reliable sources of a RAG system disappointing its team after launch.

A fourth, more subtle mistake involves permission and access control in multi-tenant or multi-user systems: treating retrieval filtering as an afterthought applied to results after they’ve already been ranked, rather than as a hard constraint enforced before ranking happens. In a system serving multiple customers or users with different access levels, a retrieval process that ranks across all data before filtering by permission — even if the filtering happens correctly before anything reaches the end user — is architecturally riskier than one where the permission boundary is enforced as the very first step, because it depends on every downstream step getting the filtering right rather than making the wrong data structurally unreachable from the start.

Finally, teams frequently underinvest in monitoring what happens after launch, treating evaluation as something done once before shipping rather than continuously. A knowledge base changes over time — documents get updated, new categories of questions start arriving, edge cases that weren’t in the original test set start showing up in traffic — and a RAG system that was carefully evaluated at launch can drift into mediocrity months later without anyone noticing, simply because nothing was in place to keep measuring it. The systems that stay reliable long-term are the ones where evaluation is treated as ongoing infrastructure, not a one-time gate before shipping.

Where RAG shows up in practice

All of these pieces — chunking, hybrid search, reranking, evaluation, and the more advanced multi-hop and agentic patterns — exist because of how many different real-world problems reduce to the same underlying shape: a large body of changing information, and a need for natural-language answers grounded in that information rather than in a model’s frozen training data. Customer support assistants use RAG to answer questions against product documentation and past ticket history that changes constantly as products evolve. Enterprise knowledge assistants use it to make an organization’s internal documentation, wikis, and policies searchable in natural language, without requiring every employee to know exactly which document or which search terms to use. Legal and compliance tools use it to search case law, contracts, and regulatory text, where retrieval precision matters because a wrong or missed citation has consequences. Coding assistants use a version of the same idea to search a codebase or documentation set for relevant functions and patterns before generating code, grounding suggestions in what a project contains rather than generic patterns learned from public code.

What ties all of these together is the same core insight this entire explanation has been built around: a language model’s usefulness on a question is bounded by what’s in front of it at the moment it answers, not by the sum total of everything it was ever trained on. Retrieval-augmented generation is, at its core, a deliberate, engineered answer to that boundary — a way of making sure the right information is in front of the model exactly when it needs it, so the answer it gives is one it was equipped to get right.