What is hallucination evaluation?

Summary slides

Why hallucination needs its dedicated evaluation practice separate from general quality checks

A response can be well-written, coherent, helpful in tone, and appropriately formatted while still containing a fabricated statistic, a citation to a paper that doesn’t exist, or a confident claim about an event that never happened, and a general quality rubric focused on helpfulness, clarity, and tone will often score that response highly precisely because those surface qualities are present even as the underlying factual content is not. This is precisely what makes hallucination such a distinctly dangerous, insidious failure mode: it doesn’t announce itself through any of the signals that typically, reliably indicate something has gone wrong, an evasive tone, an obvious error message, a visibly broken response, it hides inside output that looks, on every surface dimension a casual reader would check, exactly like a correct and trustworthy answer.

This is why hallucination evaluation has to exist as its own dedicated specialized practice with its own methods, rather than being folded entirely into a general quality score and simply trusted to surface on its own without any dedicated attention. A team that only tracks overall response quality can watch that score stay flat or even improve while the underlying hallucination rate quietly, steadily rises in the background, because the two are measuring different things entirely and a strength in one doesn’t imply any strength in the other. Treating factual reliability as a first-class, separately tracked metric in its own right, rather than an assumed, implicit byproduct of general quality, is the foundational, load-bearing premise the entire rest of hallucination evaluation builds on.

What counts as a hallucination versus an acceptable simplification or reasonable inference

Not every statement that isn’t literally, precisely true counts as a hallucination in the meaningful sense the term is used for, and drawing this line carefully matters enormously for building an evaluation practice that produces useful signal rather than noise. A reasonable simplification, describing a complex historical event in broad strokes rather than exhaustive, granular detail, is not the same failure as fabricating a fact that has no basis in actual reality at all, and a system that reasonably infers an unstated but strongly implied detail from context is behaving very differently from one that simply invents a detail with no supporting basis whatsoever. Hallucination, in the sense that matters for evaluation, specifically refers to confident, unhedged assertion of something false or unsupported, presented with the exact same certainty as a verified fact, not the ordinary, unavoidable compression that any summary or explanation necessarily, inevitably involves.

The clearest practical test distinguishes between claims that are checkable against some ground truth, whether a paper exists, whether an event happened on a date, whether a number in a source is what the model claims it is, and interpretive or synthesizing statements that don’t reduce to a single verifiable fact at all. Evaluation practices that blur this line, treating every debatable phrasing or every reasonable interpretive judgment as a hallucination alongside fabricated facts, end up with an inflated, considerably noisy hallucination rate that obscures the serious cases that truly deserve a team’s attention, which is precisely, exactly why the definitional work of scoping what counts has to happen well before any measurement can produce a number worth trusting.

How hallucination evaluation differs between closed-book and grounded generation

A system generating an answer purely from what a model learned during training, without access to any external source at the moment of answering, is being asked to recall or reconstruct information entirely from its internal parameters, and hallucination here often looks like a subtly wrong detail generated with the same fluency and confidence as a correct one, since the model has no way to distinguish between something it learned accurately and something it’s effectively guessing at based on statistical patterns in its training data that happen to resemble the shape of a correct answer. Evaluating hallucination in this closed-book setting requires checking the model’s output against external, independently verified sources, since there’s nothing in the interaction itself, no retrieved passage, no cited document, that the model’s claims can be checked against.

Grounded generation, where a model is given retrieved passages or documents and asked to answer based specifically on that provided material, changes the nature of the hallucination problem considerably, because now there is a checkable question: does the model’s answer follow from what was provided, or does it introduce claims that go beyond, or actively contradict, the source material it was given. This is sometimes called faithfulness rather than pure factual accuracy, since a model can be unfaithful to its provided source even when the underlying claim happens to be independently true, and a model can also, less obviously but just as seriously, get faithfulness backwards by cherry-picking a technically accurate detail from the source while ignoring surrounding context that would have changed how a reader should have interpreted it. Evaluating grounded hallucination specifically means checking each claim in a generated response against the actual provided source text, rather than against the broader universe of externally verified facts, which is a meaningfully more tractable problem precisely because the ground truth is right there in the retrieved context rather than scattered across the world.

How to detect hallucination at scale without exhaustive human review of every output

Human review is the most reliable way to catch hallucination, a careful reviewer with the time and expertise to check each claim will reliably outperform any automated method, but it doesn’t scale to the volume of output a production system generates, which means any practical hallucination evaluation practice has to rely heavily on automated methods for the bulk of its coverage, reserving human review for a sampled subset used both to catch what the automated methods miss and to validate that those automated methods are working correctly in the first place. This tiered structure, broad automated coverage backed by narrower but more reliable human spot-checking, is the practical foundation nearly every serious hallucination evaluation pipeline is built around.

The automated layer typically works by decomposing a generated response into individual, checkable factual claims, since evaluating an entire multi-sentence response as a single unit for hallucination produces a coarse, unhelpful signal, a response with one fabricated detail buried among several accurate ones would be marked entirely wrong or entirely right depending on an arbitrary threshold, when what’s needed is claim-level granularity that can identify exactly which assertion is the problem. Breaking a response into its component claims, then checking each one independently against a source, whether that source is a retrieved document in a grounded setting or an external knowledge base in a closed-book setting, produces considerably more actionable signal than any whole-response judgment ever could.

How automated fact-checking and claim verification pipelines work

A claim verification pipeline typically starts by extracting individual factual assertions from a generated response, a step that itself requires care, since a claim needs to be extracted with enough of its original context preserved that checking it in isolation doesn’t distort its actual meaning, a number stripped of the quantity it’s describing, or a name stripped of the claim being made about that name, becomes effectively unverifiable once decontextualized. Once claims are extracted, each one gets checked against a relevant source, either a retrieved passage in a grounded setting or a broader search or knowledge base lookup in an open setting, and the verification step itself typically uses another model, prompted specifically to judge whether a source supports, contradicts, or is simply silent on a claim, since this three-way distinction matters considerably more than a simple binary true-or-false judgment.

The distinction between contradiction and silence matters because these represent different severities of problem. A claim that a source actively contradicts is a clear, serious fabrication, the model asserted something the available evidence directly says is false. A claim the source is simply silent on is a different, somewhat less severe problem, an unsupported assertion that might still be true but wasn’t verified by anything the model had access to, and treating these two categories identically in a hallucination score conflates a dangerous failure with a more ambiguous one, which is why a well-built verification pipeline reports them as distinct categories rather than collapsing them into one aggregate hallucination percentage.

How to use a separate model as a judge to grade another model’s factuality

Using one model to grade another model’s factual claims, an approach that scales considerably better than pure human review, introduces its risk worth naming directly: the judging model’s factual knowledge has its own limits and its own biases, and a claim verification pipeline built entirely on model judgment will inherit whatever blind spots that judging model happens to carry, confidently validating a claim the judge happens to also be wrong about, or confidently rejecting a claim the judge happens to lack the knowledge to properly evaluate. This is a different risk from the non-determinism problem covered in the discussion of testing an AI system more broadly, since here the concern isn’t that the judge’s output varies from run to run, it’s that the judge can be systematically, consistently wrong in a particular direction without that consistency ever revealing the underlying problem.

Mitigating this risk means periodically validating the judging model’s verdicts against human review on a sampled basis, exactly the same discipline that matters for automated grading more broadly, and it also means being deliberate about what sources a judging model is given access to when making its verification decisions, since a judge asked to verify a claim purely from its internal knowledge, without any external source to check against, is really just comparing one model’s beliefs to another model’s beliefs rather than verifying anything against independent ground truth. Grounding the judge in retrieved, external evidence whenever possible, rather than relying on its own parametric knowledge, produces considerably more trustworthy verification results.

Why hallucination rates aren’t a single fixed number for a model

It’s tempting to think of hallucination rate as a fixed property of a model, this model hallucinates eight percent of the time, in the same way a benchmark score is often reported as a single number, but hallucination rate varies enormously depending on the task, the domain, and the way a question is asked, which means any single reported hallucination rate is only ever meaningful relative to the conditions it was measured under. A model that hallucinates rarely on well-known, heavily represented topics can hallucinate considerably more often on niche, poorly represented ones, and a model that hallucinates rarely when given relevant grounding documents can hallucinate considerably more often when asked the exact same question without that grounding present.

This variability means a hallucination evaluation practice built around a single aggregate number, measured once against a generic test set, will systematically, quietly understate the risk in exactly the domains and task types a deployment cares about, unless that evaluation is specifically, deliberately built around the actual distribution of queries that deployment will face. A team deploying a system into a domain, medical information, legal research, financial analysis, needs a hallucination evaluation built around that domain’s claims and its distinct failure patterns, not a generic hallucination benchmark that merely happens to report an encouraging headline number measured against an entirely different distribution of questions altogether.

How the nature of a task changes how much a model is likely to hallucinate

Some tasks are structurally more hallucination-prone than others, independent of how capable the underlying model otherwise is. Open-ended requests for detail, asking a model to elaborate at length on a topic, tend to produce more hallucination than narrowly scoped factual questions, simply because a longer response contains more individual claims, and each additional claim is another independent opportunity for a fabrication to slip in. Requests that ask for precise details, exact dates, exact figures, exact citations, are particularly hallucination-prone because a model under pressure to produce a-sounding answer will often generate a plausible-looking value rather than acknowledging uncertainty, since a vague or hedged answer often scores worse on a naive helpfulness metric even when it would have been the more honest response.

This means hallucination evaluation benefits from being broken down by task type rather than reported as a single blended number, tracking hallucination rate separately for narrow factual lookups, for open-ended explanatory requests, for tasks that specifically demand precise figures or citations, since a model’s risk profile looks meaningfully different across these categories even when its overall blended hallucination rate looks acceptable. A team that only tracks the blended number can miss that a system is performing perfectly well on the bulk of its traffic while hallucinating badly on the smaller category of high-precision requests that happen to carry the most consequence when they go wrong.

How confidence and calibration relate to hallucination

A model’s expressed confidence, whether it hedges a claim with “I believe” or “approximately” versus stating it flatly as settled fact, is supposed to track how reliable that claim is, but this relationship, called calibration, frequently breaks down in ways that make hallucination considerably more dangerous than it would otherwise be. A poorly calibrated model expresses the exact same confident, unhedged tone for a well-established fact and for a fabricated detail it’s effectively guessing at, which means a reader has no reliable signal from the model’s phrasing to distinguish between the two, and this absence of a reliable confidence signal is a big part of why hallucinated content is so easy to mistake for verified information.

Evaluating calibration alongside raw factual accuracy adds an important dimension to hallucination evaluation, checking not just whether a claim is true but whether the model’s expressed confidence in that claim corresponds to how likely it is to be correct, since a model that hedges appropriately on claims it’s uncertain about is considerably safer to deploy than one with an identical raw accuracy rate that expresses uniform confidence regardless of its actual underlying certainty. A well-calibrated model that occasionally hallucinates while clearly signaling its uncertainty is, in a practical sense, a safer system than a poorly calibrated one with a nominally lower hallucination rate but no reliable way for a reader to tell which claims deserve skepticism.

How hallucination evaluation should treat sourced and attributed claims differently from unsourced ones

A claim presented alongside a citation or attributed source carries an additional, separate failure mode beyond the underlying factual question: the citation itself can be fabricated, a paper that doesn’t exist, a quote misattributed to someone who never said it, an article cited with an URL that returns nothing when checked, and this failure mode deserves its dedicated verification step distinct from checking whether the underlying claim is true. A model can, in principle, state something that happens to be independently true while citing a completely fabricated source for it, and a model can also cite a source while misrepresenting what that source says, and a hallucination evaluation practice that only checks the underlying claim’s truth while ignoring citation accuracy misses a category of failure that’s particularly damaging precisely because a fabricated citation looks, to a reader who doesn’t independently check it, like the strongest possible evidence of reliability.

Verifying citation accuracy specifically means checking that a cited source exists, that it says what the model claims it says, and that the model’s characterization of that source accurately represents its actual content rather than a selectively distorted reading of it, three distinct checks that a single pass of claim verification can easily conflate into one pass-or-fail judgment unless the evaluation pipeline is deliberately structured to separate them.

How to build a hallucination benchmark dataset

A hallucination benchmark needs questions specifically designed to probe the conditions under which hallucination occurs, rather than simply reusing a general question-answering dataset and hoping hallucination happens to show up naturally in the results. This means deliberately including questions about niche, poorly represented topics where a model’s training data is thin, questions that invite unwarranted specificity, asking for an exact figure or exact date where the honest answer is uncertain, and questions where a plausible-sounding but incorrect answer is easy for a model to generate because it resembles the shape of information the model was trained on without being that information itself.

Building the ground truth for such a benchmark is itself demanding work, since every question needs a carefully verified correct answer, or in the grounded setting a clearly defined source passage the model’s answer should be checked against, and this verification work has to be done with rigor, since a benchmark whose ground truth contains errors will produce a hallucination score that’s measuring the benchmark’s mistakes as much as the model’s. The most trustworthy hallucination benchmarks are built with the same careful, adversarial mindset used in the broader practice of red-teaming, deliberately searching out exactly the conditions most likely to surface a fabrication rather than settling for whatever questions happen to be easy to write.

How hallucination evaluation connects to the broader testing and benchmarking practices already covered

Hallucination evaluation is best understood as one deep slice of the broader testing and evaluation practice that covers an AI system as a whole, focused specifically on factual reliability rather than the full range of dimensions, safety, robustness, tool use, that a comprehensive test suite has to cover. Where general system testing asks whether an output is correct, safe, and appropriate across many dimensions simultaneously, hallucination evaluation drills into one high-stakes dimension with considerably more depth and specialized tooling than a general test suite would typically apply to any single dimension on its own.

This specialization matters because hallucination has consequences disproportionate to how it might initially appear in a blended quality score, a system that hallucinates in a small fraction of its outputs can still cause serious real-world harm if those hallucinated outputs happen to concern consequential decisions, medical guidance, legal information, financial figures, which is exactly why hallucination deserves its dedicated evaluation track running alongside, rather than folded entirely into, the broader testing practice that covers a system’s behavior more generally.

How mitigation techniques get measured for their actual effect on hallucination rate

Various techniques exist for reducing hallucination, grounding a model’s answers in retrieved source material, prompting a model to explicitly cite its sources for each claim, adding an explicit verification step that checks a draft response before it’s returned, and each of these techniques needs its actual measurement of effect, since a technique that sounds like it should reduce hallucination doesn’t automatically do so in practice, and some techniques introduce new failure modes even as they reduce the original one they were designed to address. Grounding a model in retrieved documents reduces hallucination on claims the retrieved material covers but does nothing for claims that fall outside what was retrieved, and a model asked to cite sources can, without careful evaluation catching it, learn to produce plausible-looking citations that satisfy the format of sourcing without reflecting verification underneath.

This means a mitigation technique’s value can only be established by running the exact same hallucination evaluation before and after adopting it, on the exact same benchmark and the exact same distribution of queries, rather than trusting that a technique with an intuitively sound rationale will deliver the improvement its rationale suggests. Teams that adopt a mitigation technique without this before-and-after measurement often end up with a system that looks more trustworthy on the surface, more citations, more explicit grounding, while its actual underlying hallucination rate has barely moved, or has shifted into a new, less visible form the adopted technique wasn’t designed to catch.

How hallucination evaluation should account for downstream consequences, not just raw factual accuracy

Not every hallucinated claim carries the same real-world weight, a fabricated minor detail in an otherwise accurate historical summary matters considerably less than a fabricated dosage figure in a medical context or a fabricated legal precedent in a compliance-sensitive one, and a hallucination evaluation practice that treats every factual error identically, weighting a trivial fabrication the same as a consequential one, produces a score that doesn’t reflect the risk a system poses to the people relying on it. This is why the most mature hallucination evaluation practices weight claims by their actual downstream consequence, tracking not just an aggregate hallucination rate but specifically the rate of hallucination on claims that fall into a system’s high-stakes categories.

Building this consequence-weighted view requires the same kind of deliberate, domain-specific thinking that shapes how a testing bar gets set more broadly, a conversation with the actual stakeholders who understand which categories of claim, in a deployment’s context, carry serious real-world weight if they turn out to be wrong, rather than treating every fabricated detail as an equally weighted data point in a single, undifferentiated hallucination percentage.

How hallucination evaluation should account for multi-turn conversations rather than single isolated responses

Most hallucination evaluation work, by necessity of tractability, focuses on a single response to a single prompt, but usage rarely stops at one exchange, and an important category of hallucination only emerges across a conversation’s accumulating history rather than within any single turn considered in isolation. A model can correctly state a fact in one turn and then, several exchanges later, contradict that same fact once the immediate context has scrolled further back in its effective attention, or it can gradually drift a claim’s specifics across several turns of follow-up questions, each individual restatement only slightly different from the one before it, until the final version bears little resemblance to the original, accurate claim it started from. Neither of these failures shows up in a single-turn evaluation, since each individual turn, checked in isolation, might look internally consistent and reasonably well supported.

Evaluating this properly means tracking claims across an entire conversation rather than turn by turn, checking not just whether a response is internally faithful to its immediate source material but whether it remains consistent with what the same system claimed earlier in the same exchange, and flagging contradiction between turns as its distinct category of hallucination separate from a single-turn fabrication. This is considerably more expensive to evaluate than single-turn checking, since it requires maintaining and checking against an entire conversation history rather than one isolated exchange, which is part of why so much published hallucination evaluation work quietly narrows its scope to single-turn interactions even though deployed systems overwhelmingly operate across multi-turn conversations where this failure mode has room to develop.

How hallucination evaluation differs across different output modalities and formats

Hallucination is most often discussed in the context of plain text, but the same underlying failure, confident generation of something false or unsupported, shows up in structurally different ways once a system is asked to produce output in other formats, and each format requires its adapted evaluation approach rather than a single method applied uniformly everywhere. A system generating structured data, a table of figures, a formatted citation list, a set of extracted fields, can hallucinate an entire row or field that has no basis in any source, a failure that a text-focused claim extraction pipeline built around sentence-level parsing may not even be structured to catch, since the hallucinated content isn’t sitting inside a natural-language sentence at all but inside a structured value that needs its dedicated verification logic.

A system generating code presents yet another version of the same underlying problem, hallucinating a function name that doesn’t exist in the library it claims to be calling, a configuration option that was never supported, an API parameter invented to satisfy the shape of a plausible-looking call, and verifying this kind of hallucination means checking generated code against the actual interface it claims to use rather than against any kind of natural-language fact database at all. A hallucination evaluation practice built entirely around techniques designed for prose, claim extraction, sentence-level verification against retrieved passages, will systematically miss this entire category of format-fabrication unless it’s deliberately extended with format-aware verification logic built around what each output type requires to be checked correctly.

Common mistakes teams make around hallucination evaluation

Several patterns recur often enough across teams evaluating hallucination that naming them directly, explicitly is worth doing before they’re painfully discovered through a real-world incident involving a consequential fabricated claim that could have been caught earlier.

1. Treating overall response quality scores as a reliable proxy for factual reliability, missing that a well-written, fluent response can still contain confident, undetected fabrications.

2. Reporting a single blended hallucination rate without ever breaking it down by task type, domain, or the stakes of the underlying claim in question at all.

3. Evaluating hallucination purely at the whole-response level rather than the individual-claim level, missing exactly which assertion within a longer mixed response is the problem.

4. Treating a source’s silence on a claim identically to a source actively, directly contradicting that same claim, conflating two meaningfully different severities of failure.

5. Using a judging model to verify factual claims without ever validating that judge’s verdicts against careful, independent human review.

6. Letting a judging model verify claims purely, entirely from its internal, parametric knowledge rather than grounding its verification in external, retrieved evidence directly.

7. Assuming a technique that reduces hallucination in one domain or task type will reduce it equally, in a considerably different domain that wasn’t specifically tested at all.

8. Ignoring calibration entirely, focusing only on raw, blended accuracy without ever checking whether a model’s expressed confidence tracks its actual reliability at all.

9. Treating citation accuracy as automatically, implicitly guaranteed by claim accuracy, missing that a true claim can still be attached to a fabricated or considerably misrepresented source.

10. Building a hallucination benchmark from generic, off-the-shelf questions rather than deliberately probing the niche high-specificity conditions where fabrication is most likely to occur.

11. Adopting a mitigation technique based purely on its intuitive rationale without ever measuring its before-and-after effect on hallucination rate over time.

12. Weighting every hallucinated claim entirely identically regardless of its actual downstream consequence, obscuring the high-stakes failures that matter most.

13. Relying entirely, exclusively on automated detection without any human-reviewed sample to catch what the automated pipeline systematically, quietly misses each time.

14. Measuring hallucination once, a single time, against a static benchmark and treating that number as though it still applies well after the underlying model or its grounding sources have meaningfully changed since then.

15. Confusing reasonable interpretive synthesis or necessary simplification with fabrication, producing an inflated hallucination rate that buries the cases that deserve careful attention.

16. Evaluating only single-turn responses entirely in isolation, missing the contradictions and gradual factual drift that only ever emerge across a longer multi-turn conversation.

17. Applying prose-focused claim verification techniques uniformly, identically to every single output format, missing the format-distinct fabrications that show up in structured data or generated code.

What connects all seventeen of these mistakes is a single underlying pattern: treating hallucination as a single, simple, binary property that one single blended metric can adequately capture, when it is a multidimensional problem shaped by task type, domain, source grounding, claim severity, conversational history, output format, and a model’s calibration, each of which needs its dedicated measurement before a team can fully understand how much risk a system’s factual unreliability poses to the people who will end up relying on it in practice.

The deeper principle underneath all of this is that a model’s fluency and a model’s factual reliability are two entirely separate distinct properties that happen to be extremely easy to mistake for each other, since the exact same generative process that produces correct, well-supported answers also produces confident, plausible fabrications with no perceptible difference whatsoever in tone or structure, and only a deliberate, claim-level evaluation practice specifically built to check reliability rather than surface fluency can tell the two apart before a reader, trusting the confident surface of a response, has to find out the difference the hard, costly way instead.