What is RAG evaluation?

Summary slides

Why RAG systems need their dedicated evaluation approach

A standard language model evaluation asks whether a model’s output is correct given the prompt it received, but a RAG system’s actual behavior depends on something a standard evaluation doesn’t account for at all: the documents its retrieval component happened to surface before generation even began. This changes the fundamental question worth asking, since a RAG system can produce a poor final answer for two different reasons, its retrieval component failed to surface the relevant material in the first place, or its generation component had good material available but failed to use it well, and these two failure modes call for entirely different fixes, better retrieval versus better prompting or generation, that a single, undifferentiated quality score can never distinguish between.

This is precisely why RAG evaluation has to treat retrieval and generation as separate, individually measurable stages rather than evaluating the whole system purely as a black box that merely takes a query and produces a final answer. Evaluating only the final output, the way a standard LLM evaluation typically would, tells you whether the overall pipeline succeeded but gives no direct signal about where, specifically, to intervene when it didn’t, which is exactly, precisely the gap dedicated RAG evaluation methodology exists specifically to close.

The two-part structure of RAG evaluation: retrieval quality and generation quality

Retrieval quality asks whether the documents a RAG system’s retrieval component surfaced for a query are truly the relevant ones, the passages that contain the information needed to answer that query correctly, entirely independent of whatever the generation component eventually does with those particular retrieved passages. Generation quality, evaluated conditionally on whatever was retrieved, asks whether the model’s final response makes good use of that retrieved material, staying faithful to what it says rather than contradicting or embellishing it, and answering the user’s question using the relevant parts of what was retrieved rather than ignoring useful material that was available.

Separating these two stages this explicitly, deliberately matters because it lets a team diagnose failures precisely rather than merely guessing. A RAG system producing poor final answers because its retrieval component keeps missing the relevant passages needs a fix focused on the retrieval pipeline, embedding quality, chunking strategy, ranking logic, while a system whose retrieval is consistently strong but whose generation component keeps ignoring or misusing good retrieved material needs an entirely different kind of fix, focused on prompting, generation strategy, or the underlying model’s instruction-following behavior. Conflating these two stages into one undifferentiated quality score obscures exactly this distinction, which is why every mature RAG evaluation practice measures them separately before ever looking at any combined, end-to-end number.

How retrieval quality gets measured specifically

Retrieval quality is typically measured using metrics borrowed from the broader field of information retrieval, adapted to the context of a RAG pipeline. Precision asks what fraction of the documents retrieved for a query are relevant to it, penalizing a retrieval component that surfaces a lot of irrelevant noise alongside whatever useful material it does find. Recall asks the complementary question, what fraction of the relevant documents that exist in the underlying corpus got retrieved, penalizing a retrieval component that misses relevant material entirely rather than merely surfacing it alongside noise.

Ranking quality, measured through metrics that specifically account for the order documents were returned in rather than merely whether they were retrieved at all, matters considerably for RAG specifically because most RAG pipelines only pass a limited number of top-ranked documents into the generation step, which means a relevant document retrieved but ranked far down the list, below whatever cutoff the pipeline uses, contributes essentially nothing to the final answer even though it was technically retrieved. A retrieval evaluation that only checks whether relevant documents appear anywhere in an unranked result set, rather than specifically within the actual top results the generation step will see, systematically overstates how well the retrieval component is serving the rest of the pipeline.

How generation quality gets measured given a fixed set of retrieved passages

Once a fixed set of retrieved passages is held constant, generation quality evaluation asks a considerably narrower, more tractable question than open-ended generation quality does in general: given exactly this source material, did the model produce a response that makes good, appropriate use of it. This narrower framing is what makes RAG’s generation-quality evaluation considerably more tractable than evaluating open-ended generation quality broadly, since the retrieved passages themselves function as a checkable reference the generated response can be verified against, rather than requiring an evaluator to independently know the correct answer entirely from scratch.

This conditional framing also reveals a subtlety worth understanding clearly: a generation-quality evaluation conducted this way is only ever evaluating how well the model used what it was given, not whether what it was given was itself the right material in the first place, which is precisely why generation quality has to be evaluated as its separate dimension from retrieval quality rather than blended together. A model can score excellently on generation quality, faithfully and skillfully using the retrieved passages it received, while the overall system still produces a poor final answer because those particular retrieved passages were themselves the wrong ones entirely for the query being asked.

How faithfulness specifically gets measured in a RAG context

Faithfulness asks whether every claim in a generated response is supported by the retrieved source material the model was given, rather than introducing information that goes beyond, or actively contradicts, what that source material says. This is measured through the same kind of claim-level verification used in dedicated hallucination evaluation more broadly, decomposing a generated response into its individual factual assertions and checking each one against the retrieved passages, rather than judging the whole response as a single undifferentiated unit, since a response containing several faithful claims alongside one unfaithful one would be scored quite inaccurately by any method that only renders one holistic verdict.

Faithfulness in a RAG context carries a nuance worth understanding: a claim can be unfaithful either by directly contradicting the retrieved source material or by introducing information the source material simply never mentioned at all, and these represent different severities of failure that a well-built faithfulness evaluation reports separately rather than collapsing into a single score, exactly the same distinction between contradiction and silence that matters for hallucination evaluation more broadly. A model that occasionally adds a plausible but unsupported detail is failing faithfulness in a meaningfully less severe way than one that directly contradicts what its retrieved sources explicitly state.

How answer relevance is evaluated separately from faithfulness

A response can be perfectly faithful to its retrieved sources, containing no unsupported or contradictory claims whatsoever, while still failing to answer the user’s question, either by focusing on a tangential aspect of the retrieved material rather than what was asked, or by hedging so heavily and vaguely that it never delivers anything a user would find useful. Answer relevance evaluates this separate distinct dimension, asking specifically whether the generated response addresses what the user asked, independent of whether every individual claim within it happens to be well-supported by the retrieved sources.

This separation matters because faithfulness and relevance can move in opposite directions in ways a single blended metric would completely obscure: a model prompted to hedge conservatively, only ever stating what’s explicitly, directly supported by its retrieved sources, will tend to score very well on faithfulness while sometimes scoring poorly on relevance if that conservative hedging causes it to avoid directly answering a question the retrieved material does support answering with reasonable confidence. Tracking both dimensions separately, rather than trusting either one to imply the other, gives a team the actual visibility needed to tune this trade-off deliberately rather than discovering it only after users start complaining about unhelpfully evasive responses.

How context precision and context recall work as RAG metrics

Context precision and context recall are retrieval metrics specifically adapted for the RAG setting, measuring not just whether relevant documents were retrieved in some general information-retrieval sense but specifically whether the retrieved context contains what’s needed to answer the particular query at hand, and whether that useful material is concentrated near the top of the ranked results where the generation step will make use of it. Context precision asks what proportion of the retrieved context that made it into the generation step was useful for answering the query, penalizing a retrieval pipeline that pads its results with plausible-looking but ultimately unhelpful passages. Context recall asks whether all the information needed to construct a complete, correct answer was present somewhere within what got retrieved, penalizing a pipeline that retrieves reasonably relevant material while still missing some piece of information the query requires.

These two metrics, evaluated together, catch a failure pattern that neither one alone reveals clearly: a retrieval pipeline that achieves excellent precision by retrieving a small number of highly relevant passages can still have poor recall if the query required synthesizing information spread across several passages the pipeline failed to surface all of and a pipeline achieving excellent recall by retrieving broadly can have poor precision if it buries useful material amid a large amount of retrieved noise. Tracking both together, rather than either alone, gives a considerably more complete picture of exactly how a retrieval pipeline is failing when it does fail.

How to build a RAG evaluation dataset with ground-truth documents

Building a trustworthy RAG evaluation dataset requires more than a list of questions with correct answers, it requires knowing, for each question, which documents in the underlying corpus contain the information needed to answer it correctly, ground truth that a general question-answering benchmark typically doesn’t provide at all since it was never built with a retrieval corpus in mind. This means a useful RAG evaluation dataset has to be built specifically against the actual corpus a system retrieves from, identifying, for each evaluation question, the exact passages within that corpus that constitute a correct, complete answer, work that’s considerably more labor-intensive than simply writing questions and their expected answers in isolation.

This corpus-requirement is part of why generic, publicly available RAG benchmarks provide a useful early signal but rarely substitute for an evaluation dataset built against an organization’s actual retrieval corpus, since a system’s performance depends heavily on the characteristics of that corpus, its size, its organization, the quality and consistency of its underlying content, none of which a generic benchmark built against an entirely different corpus can capture. A team evaluating a RAG system deployed against its proprietary or highly specialized corpus of documents needs its dedicated evaluation dataset built specifically against that corpus to get results that mean something for its deployment.

How end-to-end RAG evaluation differs from evaluating the two components separately

Evaluating retrieval and generation separately provides essential diagnostic precision, but it doesn’t fully substitute for also measuring the complete pipeline’s actual end-to-end output quality, since the two stages can interact in ways that separate, isolated evaluation doesn’t fully capture, a generation component that performs excellently when evaluated against clean, ideal retrieved passages in isolation can behave meaningfully differently once it’s working with the imperfect, sometimes irrelevant passages its retrieval component realistically surfaces in production, a gap that only becomes visible through end-to-end evaluation rather than component-level testing alone.

The most complete RAG evaluation practices run both layers together: component-level evaluation to diagnose exactly where a problem originates once one is detected, and end-to-end evaluation, measuring the final response’s overall quality against real or realistic queries, to catch the kind of interaction effects that only emerge once the full pipeline is operating as a connected whole rather than as two components tested independently of each other. Relying purely on component-level metrics risks missing an end-to-end quality problem that neither component’s isolated score would individually reveal, while relying purely on end-to-end metrics risks losing the diagnostic precision needed to fix a problem once one is found.

How chunk size and retrieval configuration choices show up in evaluation results

A RAG pipeline’s configuration choices, how source documents get split into retrievable chunks, how many chunks get retrieved per query, how those chunks get ranked and filtered before reaching the generation step, have a direct, often substantial effect on evaluation results, and understanding this connection turns RAG evaluation from a passive measurement exercise into an active tool for tuning the pipeline’s configuration. Chunks that are too small can fragment information across multiple pieces in a way that hurts context recall, since a single retrieved chunk might not contain a complete, self-sufficient answer even when the surrounding, un-retrieved context would have. Chunks that are too large can hurt context precision, since a large chunk retrieved for being broadly relevant often carries a lot of irrelevant surrounding material along with whatever useful information it contains.

Running RAG evaluation systematically across a range of different chunking and retrieval configuration choices, rather than evaluating only a single fixed configuration, lets a team see how these choices trade off against each other in measured, concrete terms rather than relying on intuition or generic best-practice guidance that may not hold for a particular corpus and query distribution. This kind of configuration sweep, evaluating retrieval and generation quality across several different chunking and retrieval settings, is one of the most directly actionable uses of RAG evaluation, since its output maps immediately onto a concrete engineering decision rather than remaining an abstract quality score with no clear next step attached to it.

How RAG evaluation should treat cases where no relevant document exists

Some queries a RAG system receives have no correct answer available anywhere in the underlying corpus, either because the information simply isn’t covered by any document the system has access to, or because the query itself falls outside what the corpus was ever built to address, and a well-built RAG evaluation practice has to specifically account for this category rather than assuming every evaluation query has a retrievable correct answer waiting somewhere in the corpus. The correct, desired behavior in this case is for the system to recognize the absence of relevant material and communicate that honestly, rather than generating a plausible-sounding but ultimately fabricated answer stitched together from whatever tangentially related material its retrieval component happened to surface anyway.

Evaluating this behavior means deliberately including queries with no answer in the corpus as their distinct category within a RAG evaluation dataset, scoring a system specifically on whether it correctly recognizes and honestly communicates this absence rather than confabulating an answer, since a RAG evaluation practice that only ever tests queries with a known, retrievable correct answer will never surface this and consequential failure mode, one of the more dangerous behaviors a RAG system can exhibit precisely because a fabricated answer stitched together from irrelevant retrieved material can look, to a reader who doesn’t check the underlying source, just as confident and well-supported as a correct one.

How multi-hop and multi-document RAG evaluation differs from single-passage retrieval evaluation

Many queries can’t be answered from a single retrieved passage at all, they require synthesizing information spread across several separate documents, sometimes requiring the system to use information found in one retrieved passage to inform what it should search for next, a pattern called multi-hop retrieval that introduces evaluation challenges single-passage retrieval evaluation was never built to address. A retrieval evaluation built purely around checking whether a single relevant passage was retrieved for a query will systematically miss cases where the correct answer requires combining several distinct passages, none of which alone constitutes a sufficient, complete answer on its own.

Evaluating multi-hop RAG performance well means building evaluation questions that specifically require this kind of synthesis across documents, and scoring both whether all the necessary passages were retrieved across however many hops the query required, and whether the generation component successfully combined that spread-out information into one coherent, accurate final answer rather than only ever using whichever single passage happened to be ranked highest. This considerably more demanding evaluation setup reflects a more demanding capability, since multi-hop reasoning over retrieved material is meaningfully harder for a RAG system to get right than simple single-passage lookup, and a system that performs excellently on single-passage evaluation queries can still perform considerably worse once multi-hop, multi-document queries enter the picture.

How RAG evaluation connects to hallucination evaluation as a complementary practice

RAG evaluation and dedicated hallucination evaluation share technical overlap, both rely on claim-level verification checking whether a generated response’s individual assertions are supported by some external source, but they answer different scoped questions and serve different purposes within a broader evaluation practice. Hallucination evaluation, in its general form, checks a model’s factual reliability broadly, sometimes against retrieved sources in a grounded setting and sometimes against a model’s general knowledge in a closed-book setting entirely disconnected from any retrieval step. RAG evaluation’s faithfulness dimension is a narrower application of that same underlying technique, scoped tightly to the particular retrieved passages a RAG pipeline surfaced for a query.

Because of this overlap, a team building a RAG evaluation practice from scratch can often reuse much of the claim-verification infrastructure that a more general hallucination evaluation practice already relies on, extending it specifically to check against the retrieved passages a RAG pipeline surfaces rather than against a broader, more general external source. Conversely, a RAG evaluation practice that only ever checks faithfulness against its retrieved passages, while never separately tracking the broader hallucination risk a model carries independent of retrieval, can miss cases where a model’s underlying tendency toward fabrication persists even when relevant correct source material was available and retrieved successfully.

How to run RAG evaluation continuously as a retrieval corpus and underlying model change

A RAG system’s underlying corpus rarely stays fixed, new documents get added, old ones get updated or removed, and each change to that corpus is an opportunity for retrieval quality to shift in ways a one-time evaluation, run before the corpus started changing, would never catch. Similarly, an underlying model update, the same kind of change covered in the broader discussion of ongoing agent evaluation, can shift generation quality even when retrieval itself remains completely unchanged. This means RAG evaluation, like broader agent evaluation, has to run continuously rather than as a one-time pre-launch check, specifically re-evaluating retrieval quality whenever the underlying corpus changes meaningfully and re-evaluating generation quality whenever the underlying model changes.

Building this continuous evaluation practice specifically for RAG means maintaining an evaluation dataset that stays representative of the corpus’s current, evolving content, since an evaluation dataset built against an earlier version of a corpus can become progressively less representative as that corpus grows and changes, quietly leaving newer content without any evaluation coverage at all. A team that treats its RAG evaluation dataset as a fixed artifact, built once and never revisited even as the underlying corpus it was built against keeps changing considerably, will find its evaluation coverage increasingly disconnected from what the system is retrieving from and generating against in current production use.

How RAG evaluation should account for retrieval latency and cost, not just retrieved quality

A retrieval configuration that achieves excellent precision and recall by searching an enormous number of candidate passages, applying several successive rounds of reranking, and pulling in a considerably larger context window than a leaner configuration would use, is not delivering the same value as a leaner configuration achieving comparable quality more efficiently, and a RAG evaluation practice that tracks retrieval and generation quality alone, without any accompanying view of latency and computational cost, will systematically miss this important trade-off. This matters considerably more once a RAG system is serving production traffic at scale, where a marginal quality improvement purchased through a considerably more expensive retrieval pipeline, more candidate passages searched, more reranking stages applied, a larger context window passed to generation, may simply not be worth the cost for a deployment, even though a quality-only evaluation would report it as a straightforward win.

Tracking efficiency alongside retrieval and generation quality also surfaces a category of degradation that a pure quality metric would never reveal on its own: a retrieval pipeline that’s technically still achieving strong precision and recall but has quietly grown considerably more expensive over time, searching a larger candidate pool, applying more reranking passes, retrieving more chunks than strictly necessary to reach the same quality bar it used to reach more efficiently, a costly drift that only becomes visible once latency and cost are tracked as their dedicated metrics running alongside quality rather than left entirely unmeasured.

How to evaluate a RAG system’s behavior when retrieved passages conflict with each other

Corpora are rarely perfectly internally consistent, different documents can disagree, an outdated document contradicting a more recent one, two sources describing the same underlying fact with different figures, and a RAG system’s retrieval component can easily surface both sides of a conflict within the same set of retrieved passages for a single query. This creates a distinct evaluation challenge beyond ordinary faithfulness checking, since a response can be faithful to one of the retrieved passages while simultaneously contradicting another equally retrieved one, and a faithfulness check built around verifying claims against retrieved source material in the aggregate, without specifically accounting for this kind of internal conflict, can produce a misleadingly reassuring score even when the underlying retrieved context was contradictory.

Evaluating this behavior well means deliberately including queries in a RAG evaluation dataset where the underlying corpus is known to contain conflicting information, and scoring a system specifically on whether it correctly recognizes and surfaces that conflict to the user, rather than silently picking one side of a disagreement and presenting it with unwarranted, single-answer confidence, or worse, blending the conflicting figures together into a fabricated middle-ground answer that matches neither actual source. A RAG evaluation practice that never tests this scenario will miss one of the more subtle and consequential failure modes a retrieval-grounded system can exhibit, since a confidently stated answer drawn from a disputed source can be just as misleading to a user as an answer with no source support at all.

How RAG evaluation should treat recency and time-sensitive information

A corpus that includes documents written at different points in time introduces an evaluation challenge beyond ordinary conflict between sources: a query about a fact that has changed over time, a figure that was accurate when an older document was written but has since been updated, a policy that was correct at one point but has since changed, requires a retrieval component that specifically favors the most current, accurate source rather than simply the most semantically similar one, and a generation component that correctly reflects that currency rather than treating every retrieved passage as equally authoritative regardless of how outdated it is. Standard retrieval and faithfulness metrics, built around checking relevance and support in the abstract, don’t inherently account for this temporal dimension at all, and a system can score well on both while still confidently surfacing stale, outdated information as though it were current.

Evaluating this behavior well means deliberately including time-sensitive queries in a RAG evaluation dataset, queries whose correct answer specifically depends on retrieving and prioritizing the most current available source among several documents covering the same underlying topic at different points in time, and scoring a system specifically on whether it correctly surfaces and reflects that most current information rather than an older, superseded version that happens to be an equally strong semantic match for the query. A RAG system deployed against a corpus that updates regularly, policy documents, product specifications, pricing information, needs this evaluation category built into its practice from the start, since the consequences of confidently surfacing outdated information as though it were current can be just as serious as the consequences of any other faithfulness failure covered elsewhere in this discussion.

Common mistakes teams make around RAG evaluation

Several patterns recur often enough across teams evaluating RAG systems that naming them directly, explicitly is worth doing before they lead to a costly gap in understanding how a deployed RAG pipeline is truly performing in practice.

1. Evaluating only the final generated output, missing whether a poor result originated from retrieval failure or generation failure, and losing the diagnostic precision that separate component evaluation provides.

2. Measuring retrieval quality without accounting for ranking, treating a relevant document retrieved but ranked far below the pipeline’s actual cutoff as equivalent to one surfaced near the top.

3. Conflating faithfulness and answer relevance into a single blended score, missing cases where a faithful response still fails to answer the question being asked.

4. Treating a generic, publicly available RAG benchmark’s results as directly predictive of performance against an organization’s proprietary retrieval corpus.

5. Building an evaluation dataset without any queries that have no correct answer anywhere in the corpus, missing the dangerous fabrication risk this category is meant to surface.

6. Evaluating retrieval purely on single-passage lookup, missing the harder, considerably more demanding multi-hop, multi-document synthesis many queries require.

7. Never running a systematic sweep across different chunking and retrieval configurations, missing directly actionable signal about how those choices trade off in practice.

8. Treating faithfulness contradiction and faithfulness silence as the exact same severity of failure, obscuring the more dangerous, more consequential of the two categories.

9. Running RAG evaluation only once before launch and never revisiting it as the underlying corpus and retrieval configuration keep changing over real time.

10. Evaluating generation quality only against clean, ideal retrieved passages in isolation, missing how the model behaves once facing its retrieval component’s imperfect output.

11. Folding RAG faithfulness checking entirely into a generic quality rubric rather than running the dedicated claim-level verification the dimension specifically requires.

12. Letting an evaluation dataset grow stale relative to an actively evolving corpus, leaving newer content with essentially no evaluation coverage at all whatsoever.

13. Relying purely, entirely on component-level metrics without ever measuring end-to-end quality, missing interaction effects that only emerge once the full pipeline runs together.

14. Assuming strong context precision automatically, implicitly implies equally strong context recall, missing that a retrieval pipeline can excel at one while quietly failing at the other.

15. Building RAG evaluation entirely in isolation, without ever connecting its faithfulness metrics to the organization’s broader hallucination evaluation practice.

16. Tracking retrieval and generation quality without any accompanying view of latency and computational cost, missing when a retrieval pipeline quietly becomes considerably more expensive over real time.

17. Never testing how a system handles conflicting retrieved passages, missing a failure mode where confident, single-answer output masks an underlying source disagreement.

18. Ignoring recency and time-sensitivity entirely, letting a system confidently surface outdated, superseded information as though it were still the current, accurate answer.

What connects all eighteen of these mistakes is a single underlying pattern: evaluating a RAG system as though it were a single, undifferentiated black box rather than a two-stage pipeline whose retrieval and generation components can each fail independently, in their ways, and each require their dedicated, specifically tailored evaluation methodology before an aggregate, end-to-end quality score can be trusted to mean what it appears to mean.

The deeper principle underneath all of this is that a RAG system’s final output quality is never a single, unified property, it’s the joint, compounding result of several separate processes, finding the right material, using it well, respecting its actual recency, and honestly surfacing disagreement when the underlying corpus itself contains one, and only an evaluation practice built specifically to measure each of these processes independently, rather than collapsing them into one blended score, can tell a team where its system is succeeding, where it’s failing, and precisely what to concretely fix once a failure has been found.