What is RAG evaluation?

Summary slides

Why RAG evaluation needs to separate retrieval quality from generation quality

A RAG system’s final answer depends on two distinct stages working correctly, retrieval finding truly relevant content, and generation using that content correctly to produce an accurate, well-grounded response, and these two stages can fail independently of each other in ways that look identical from the outside. A poor final answer might trace back to retrieval missing the genuinely relevant passage entirely, or it might trace back to retrieval finding exactly the right content while generation still failed to use it correctly, producing an answer that ignores or misreads what was actually retrieved.

Evaluating only the final output without this decomposition makes it considerably harder to diagnose which stage needs improvement, a team seeing poor final answers might invest in improving generation when the real problem was retrieval, or vice versa, wasting effort on the wrong part of the system entirely. This is why RAG evaluation frameworks measure retrieval and generation quality as distinct, separately trackable metrics rather than relying purely on end-to-end answer quality alone.

How context relevance measures whether retrieval found the right content

Context relevance evaluates whether the passages a RAG system’s retrieval step surfaced are genuinely relevant to the query, independent of whatever the generation step eventually did with them. This connects directly to the retrieval evaluation approaches covered throughout this collection’s discussion of retrieval pipelines and embedding similarity, but specifically applied within the context of a complete RAG system, checking whether the actual passages that made it into the model’s context were the ones genuinely needed to answer the specific question being asked.

Measuring context relevance in isolation, separate from whether the final generated answer turned out correct, is what lets a team catch a retrieval problem even in cases where generation happens to produce a passable answer despite receiving poor context, perhaps by falling back on the model’s own general knowledge rather than the retrieved content, a situation that looks superficially fine but represents exactly the kind of hidden, uncontrolled reliance covered throughout this collection’s broader discussion of hallucination and verification.

How faithfulness measures whether the generated answer stays grounded in retrieved content

Faithfulness, sometimes called groundedness, evaluates whether a generated response’s claims are supported by the content that was retrieved, rather than the model introducing information from its own general training knowledge or, worse, fabricating claims that aren’t supported by anything in the retrieved context at all. This is a distinct failure mode from poor retrieval, a RAG system can retrieve genuinely relevant, correct content and still produce an unfaithful answer if the generation step doesn’t ground its response in that content reliably.

Measuring faithfulness typically involves checking each claim in a generated answer against the retrieved context directly, verifying that the claim is supported by something present in that context rather than simply being plausible or generally consistent with the topic. This connects directly to the broader discussion of hallucination covered elsewhere in this collection, faithfulness evaluation is specifically the RAG-focused version of checking whether a model’s output is grounded in verifiable source material rather than confidently generated from pattern-based guessing.

How answer relevance measures whether the response actually addresses the question

Answer relevance evaluates whether a generated response actually addresses what the user asked, independent of whether it’s faithfully grounded in the retrieved context, a response can be perfectly faithful to retrieved content while still failing to directly answer the specific question asked, perhaps by providing accurate but tangential information rather than the precise answer needed. This is a third, genuinely distinct failure mode from both retrieval quality and faithfulness, a RAG system can retrieve the right content and generate a faithful response using it, while still producing an answer that doesn’t actually satisfy what the user was looking for.

This three-way decomposition, context relevance, faithfulness, answer relevance, gives a considerably more diagnostic picture of RAG system quality than any single metric alone could provide, a team can identify specifically whether a quality problem traces back to retrieval, to ungrounded generation, or to generation that’s grounded but off-target, and address the actual root cause directly rather than making broad, undirected changes across the entire pipeline.

Why RAG evaluation benefits from using a language model to judge quality automatically

Manually judging retrieval relevance, faithfulness, and answer relevance for every test case doesn’t scale well as a RAG system evolves and needs frequent re-evaluation, which is why many RAG evaluation approaches use a separate language model specifically to judge these dimensions automatically, checking whether a claim is supported by retrieved context, or whether an answer addresses a given question, at a speed and scale manual human review can’t match. This connects to the broader pattern of using models to evaluate other models’ outputs covered throughout this collection’s discussion of AI native testing, applied here specifically to the distinct dimensions of RAG quality this article has described.

This automated approach carries its own genuine limitation worth acknowledging directly, a model used to judge faithfulness or relevance can itself make judgment errors, and relying entirely on automated evaluation without any human spot-checking risks building false confidence in a RAG system’s quality based on an evaluation mechanism that has its own, separate failure modes worth periodically validating against actual human judgment.

Why RAG evaluation needs to happen continuously, not just at initial deployment

A RAG system’s underlying content collection, embedding model, and even the distribution of user queries it receives all change over time, and evaluation performed once during initial development doesn’t guarantee the same quality holds once a system has been running in production for a while, connecting directly to the broader discussion of ongoing calibration and monitoring covered throughout this collection. A RAG system that scored well during initial testing can gradually degrade as its underlying content grows stale, as new kinds of queries arrive that weren’t represented in the original test set, or as the embedding model or retrieval configuration drifts from what was originally validated.

This is why mature RAG deployments treat evaluation as an ongoing operational discipline rather than a one-time gate passed before launch, running the context relevance, faithfulness, and answer relevance metrics this article has described continuously against a sample of real production traffic, catching degradation before it accumulates into a noticeably worse user experience rather than discovering it only after users have already been affected.

Common mistakes teams make around RAG evaluation

1. Measuring only end-to-end answer quality without decomposing it into retrieval, faithfulness, and relevance, making it difficult to diagnose which part of the pipeline actually needs improvement.

2. Treating a faithful, well-grounded answer as automatically good, missing that it can still fail to actually address what the user was asking.

3. Relying entirely on automated, model-based evaluation without any periodic human validation of that evaluation mechanism’s own accuracy.

4. Treating RAG evaluation as a one-time gate before launch rather than an ongoing discipline that catches gradual degradation in production.

5. Evaluating faithfulness using test cases that don’t reflect the genuine complexity and ambiguity of real user queries, missing failure modes that only emerge with realistic, messy production traffic.

What connects these mistakes is treating RAG evaluation as a single, simple pass-or-fail judgment rather than a multidimensional, ongoing discipline that isolates specific failure modes and tracks them continuously, a RAG system’s real quality depends on several distinct things going right together, and evaluation that doesn’t separate and continuously monitor each of them misses exactly the diagnostic detail needed to actually improve the system where it genuinely needs improvement.

The deeper point about RAG evaluation is that a retrieval-augmented system’s trustworthiness depends on more than whether its answers sound plausible, it depends on whether those answers are actually grounded in genuinely relevant, correctly retrieved content, and building that trust requires the kind of decomposed, continuously measured evaluation this article has described, rather than trusting a system simply because its outputs read fluently and confidently on the surface.