What is AI evaluation?

Summary slides

Why AI evaluation exists as its broad discipline

Traditional software has a relatively clear notion of correctness, a program either produces the output its specification calls for or it doesn’t, and verifying that correctness is largely a matter of careful, systematic testing against known inputs and expected outputs. AI systems, particularly those built around large language models, break this clean relationship between specification and verification, since the systems themselves are trained rather than explicitly programmed, their behavior emerges from patterns in data rather than from rules a developer wrote directly, and their outputs are often open-ended rather than reducible to a single correct answer. This shift is exactly, precisely why AI evaluation had to emerge as its dedicated, distinct discipline rather than simply, entirely inheriting the methods and assumptions of conventional software testing wholesale, largely unchanged.

The discipline exists because the stakes of getting this measurement wrong keep steadily rising as these systems take on considerably more consequential real-world roles, informing medical decisions, handling financial transactions, acting with autonomy on an actual person’s behalf, and an organization deploying a system into any of these consequential roles needs well-grounded confidence that the system performs the way it’s expected to, confidence that can only come from a deliberate, rigorous measurement practice rather than from mere intuition, anecdote, or the system’s confident-sounding, plausible output. AI evaluation, in this broad sense, is the field that has grown up specifically, deliberately to provide that grounded confidence, rather than leaving it to chance or to whatever intuition a team happened to already have.

What distinguishes evaluation from adjacent disciplines like testing and monitoring

Evaluation, testing, and monitoring are related but distinct activities, and conflating them leads to gaps in how an organization understands its system’s behavior. Testing, in the narrower, more technical sense, checks a system against a defined, fixed set of scenarios before a change ships, producing a binary pass-or-fail verdict meant specifically to catch regressions before they reach users. Monitoring watches a live, deployed system’s operational health, uptime, latency, error rates, the kind of infrastructure-level signals that indicate whether a system is running correctly at a purely technical level, entirely independent of whether its actual outputs are good ones.

Evaluation sits between and alongside both of these, asking a different, distinct question from either one: not merely whether a system passes some predefined test, and not merely whether it’s technically running without visible errors, but whether its actual behavior and output quality meet the bar that matters for the actual people relying on it, measured continuously against live usage rather than checked only once before shipping and never again. A system can pass every single pre-launch test and show a perfectly healthy, green operational dashboard while still producing outputs a careful, deliberate evaluation would specifically flag as inadequate, which is precisely, exactly the gap evaluation, as its broad, distinct discipline exists specifically to close.

The major dimensions AI evaluation spans

A comprehensive evaluation practice tracks several distinct dimensions of a system’s behavior, and no single dimension, however carefully measured, can stand in for the others. Capability asks what a system is able to do, its raw, underlying ability to answer questions correctly, solve problems, and generate useful output, the dimension most directly measured by traditional benchmarking. Reliability asks how consistently a system performs that same capability across repeated attempts and varying conditions, since a system that succeeds most of the time but fails unpredictably carries risk a pure capability score would never reveal on its own.

Safety asks whether a system avoids causing harm refusing dangerous requests, avoiding the generation of harmful content, and respecting the boundaries it’s been given from the start. Efficiency asks how much a level of quality costs, in compute, in time, in the actual number of steps a task requires to complete. Alignment, a somewhat considerably broader and more actively contested dimension, asks whether a system’s actual behavior reflects the intentions and values of the people deploying it, rather than pursuing some narrower, overly literal interpretation of its instructions that technically, superficially satisfies the letter of a request while violating its actual spirit entirely. A thorough, comprehensive evaluation practice tracks all of these distinct dimensions in parallel, since strength in one says relatively little about strength in any of the others.

How evaluation differs across a system’s lifecycle stages

Evaluation isn’t a single, fixed activity performed once and then set aside, it changes shape considerably across the different, distinct stages of a system’s life. During model development, evaluation focuses heavily, primarily on benchmarking, comparing candidate models and different training approaches against standardized capability measures to guide decisions about which direction to pursue next. Before actual deployment, evaluation shifts toward system-testing, carefully checking that the particular application built around a chosen model behaves correctly for its intended use cases, tools, and constraints, distinct from the underlying model’s general capability.

After deployment, evaluation becomes an ongoing, continuous practice, tracking production behavior against usage rather than against any fixed, predetermined set of scenarios, since usage inevitably surfaces situations no pre-deployment process fully anticipated. Each of these distinct stages asks a different question suited specifically to what’s knowable and actionable at that particular point in a system’s life, and an organization that only ever invests evaluation effort at one single stage, treating a strong pre-deployment test suite as sufficient assurance forever after, or relying purely on post-deployment monitoring without ever having compared candidate models carefully beforehand, leaves gaps precisely at the stages it neglected.

How evaluation methodology has evolved over time

Early evaluation of language models leaned heavily on simple, automatically, easily scored benchmarks, multiple-choice knowledge tests, exact-match question answering, formats chosen largely because they could be scored without any human judgment or sophisticated automated grading required at all. As models grew considerably more capable over time, these simpler benchmarks increasingly, steadily saturated, every single leading model clustering near the maximum score, which pushed the field toward considerably more demanding formats open-ended generation scored by careful, detailed rubrics, multi-step reasoning problems, and eventually, over time, toward evaluating models embedded in interactive, tool-using agent loops rather than merely answering isolated, static questions.

This evolution reflects a considerably broader pattern worth understanding on its own: evaluation methodology has to keep pace with the systems it evaluates, and a methodology well-suited to an earlier generation of considerably less capable models eventually just stops providing much useful signal once the field’s underlying capability has moved meaningfully past what that methodology can discriminate. The current emphasis on evaluating agents, tracking hallucination specifically, and running continuous, production-grounded evaluation all represent the field’s ongoing adaptation to systems that are considerably more capable, more autonomous, and more deeply embedded in consequential decisions than the systems the earliest benchmarks were originally built to measure.

The role of human judgment throughout evaluation

Human judgment remains, across every stage and every dimension of AI evaluation, the most reliable single source of assessment, since a careful, well-briefed human reviewer can catch nuance, context, and appropriateness in a way no automated method fully replicates. This is why human review persists as a core component of evaluation practice even as automated methods have grown considerably more sophisticated, used both directly, evaluating samples of output against carefully constructed rubrics, and indirectly, calibrating and validating the automated methods that carry the bulk of evaluation volume at production scale.

The practical challenge human judgment poses is that it doesn’t scale to the volume of output modern AI systems generate, which means the skill in building a mature evaluation practice lies in deploying human review deliberately and strategically, at the points where it adds the most value, rather than either abandoning it entirely in favor of pure automation or attempting to apply it uniformly at a volume no human review process could ever realistically sustain.

The role of automated and model-based evaluation

Automated evaluation, whether through simple rule-based checks or through a separate model prompted to grade another model’s output, is what makes evaluation tractable at production scale, extending coverage to volumes of output no human review practice could match. This scalability comes with a structural trade-off: an automated grader, particularly a model-based one, carries its own biases and blind spots, and its judgments can drift away from what a careful human reviewer would conclude without that drift becoming visible unless it’s periodically checked against human calibration.

The most mature evaluation practices treat automated methods not as an independent, self-sufficient source of truth but as a scaling mechanism whose outputs are only as trustworthy as the human calibration process validating them, a discipline that applies whether the automated method is grading a benchmark’s open-ended responses, verifying a claim’s factual accuracy, or scoring an agent’s multi-step trajectory against a defined rubric. Automated evaluation without this ongoing calibration risks confidently reporting numbers that look rigorous while measuring something meaningfully different from quality.

How evaluation results get used to make decisions

Evaluation that never influences a decision is, in a practical sense, wasted effort regardless of how methodologically rigorous it happens to be, and the value of an evaluation practice ultimately depends on whether its findings translate into choices, which model to build on, which change to ship, which feature to hold back, rather than existing purely as a reporting exercise disconnected from actual action. This means evaluation results need to reach the people with the authority to act on them, translated into terms those people can use, and the organizations that get the most value from evaluation build this connection deliberately rather than assuming it will happen automatically once the measurement work is done.

This decision-relevance also shapes what evaluation should measure in the first place, since a metric that’s rigorous but disconnected from any decision a team will ever make is considerably less valuable than a somewhat rougher metric that’s tied to a choice the organization regularly has to make, and building evaluation practice backward from the actual decisions it needs to inform, rather than forward from whatever happens to be easy to measure, tends to produce a considerably more useful practice overall.

The tension between standardized and custom evaluation

Standardized, widely shared benchmarks offer advantages, comparability across different models and organizations, established methodology that doesn’t need to be built from scratch, a shared reference point the broader field can discuss and improve collectively over time. But a standardized benchmark, by its very nature, measures general capability rather than performance on any organization’s particular use case, and the gap between strong performance on a general benchmark and strong performance on a deployment’s actual traffic can be considerably larger than an organization relying purely on standardized results might expect.

Custom, task-specific evaluation closes this gap but sacrifices the comparability and shared methodology a standardized benchmark offers, requiring dedicated investment to build well, and the most effective evaluation practices use both in a deliberately layered way, standardized benchmarks for broad, early filtering among candidate models and approaches, custom evaluation for the deeper validation that a particular deployment needs before and after it goes live. Treating either one as a complete substitute for the other, relying purely on general benchmarks without ever building anything custom, or building custom evaluation while ignoring the broader field’s established standardized methods entirely, leaves gaps in an organization’s overall understanding of its system.

How the sub-practices of evaluation fit together as parts of a broader discipline

The many evaluation practices, benchmarking a candidate model before it’s chosen, testing a system’s behavior before a change ships, tracking hallucination with dedicated claim-level verification, running continuous evaluation of an already-deployed agent’s production behavior, each address a distinct part of the broader question AI evaluation as a whole exists to answer, and none of them, on its own, provides complete coverage. Benchmarking answers the earliest question, which model or approach is worth building on in the first place. Testing answers the next one, does this system, built on that chosen foundation, behave correctly before it reaches users. Continuous evaluation answers the question that persists indefinitely once a system is live, is this system still behaving well against the evolving usage it’s facing right now. And hallucination evaluation drills into one particular, high-stakes dimension, factual reliability, with more depth than any of the broader practices alone would provide.

Understanding how these pieces fit together prevents a common, costly mistake, treating strong performance in one of these more practices as though it settled the broader question evaluation as a whole is meant to answer. A model that benchmarks exceptionally well hasn’t necessarily been tested against a deployment’s particular requirements. A system that passed every pre-launch test hasn’t necessarily been evaluated against the drift and edge cases that only emerge once it’s live. Each piece is necessary, and treating any single piece as sufficient on its own leaves a gap in an organization’s overall picture of how its system is performing.

How evaluation itself is a maturing field with its open problems

AI evaluation is not a settled, fully solved discipline, and several of its core methodological challenges remain unresolved even as the field has matured considerably from its earliest, simplest benchmarks. Contamination, the risk that a benchmark’s answers have leaked into a model’s training data, remains a persistent problem without any fully satisfying general solution, since keeping evaluation data entirely private trades away the transparency and reproducibility that make evaluation results independently trustworthy in the first place. Model-based grading, essential for scaling evaluation to production volume, still carries the unresolved risk that the grading model’s judgment diverges from human assessment in ways that are hard to detect without continuous, deliberate calibration effort.

These open problems matter for how a practitioner should engage with the field’s current state, treating today’s evaluation methodology as the best currently available approach rather than as a finished, fully reliable science, and staying attentive to the field’s evolving best practices rather than assuming a method that was considered rigorous several years ago remains equally rigorous today. A discipline this actively evolving rewards practitioners who stay closely engaged with how evaluation methodology itself is changing, not just with how the systems being evaluated are changing.

Organizational and governance aspects of evaluation

Building effective AI evaluation requires more than sound technical methodology, it requires organizational structure: clear ownership of who’s responsible for running evaluation and acting on its findings, a defined process for escalating serious findings rather than letting them sit in a routine report nobody specifically owns, and a governance structure that gives evaluation findings weight in actual deployment decisions rather than treating them as an afterthought that gets acknowledged but rarely changes anything. Without this organizational layer, even a technically excellent evaluation practice tends to atrophy into a reporting exercise that documents problems without ever driving the changes that would fix them.

This governance dimension becomes considerably more important as AI systems take on higher-stakes roles, since an organization deploying a system into a consequential context needs evaluation findings to carry authority, potentially including the authority to block or delay a deployment when evaluation surfaces a serious concern, rather than functioning purely as advisory input that a deployment decision can proceed around regardless of what it found. Building this authority into an organization’s evaluation practice is as much a governance and organizational design question as it is a technical one.

How evaluation connects to trust and to real-world adoption

The ultimate purpose of AI evaluation, beneath all its methodologies and dimensions, is enabling well-founded trust, trust that a system will behave the way it’s expected to, held not just by the organization deploying it but by the people who will be affected by its behavior, whether that’s an end user relying on its output or a broader public affected by its wider deployment. Evaluation that’s rigorous but never communicated to the people who need to trust its results doesn’t fully serve this purpose, since trust requires transparency about what was measured and how, not just a confident assurance that measurement happened.

This connection between evaluation and trust matters considerably for the pace and scope of real-world AI adoption more broadly, since organizations, regulators, and the general public are, reasonably, more willing to extend trust to systems whose evaluation practices are rigorous and transparently documented than to systems whose capability is simply asserted without any visible, credible measurement behind that assertion. A mature evaluation practice, in this sense, serves a function considerably broader than internal quality assurance, it’s part of what makes broader, responsible AI adoption possible in the first place.

Where evaluation practice is heading

As AI systems take on more autonomous, longer-running, and more consequential roles, evaluation practice is visibly shifting to keep pace, moving from single-turn, static benchmarks toward evaluation of extended, multi-step agent behavior, from one-time pre-launch checks toward continuous, always-on production evaluation, and from purely capability-focused measurement toward an integrated view spanning capability, safety, reliability, and alignment together rather than as separate, disconnected concerns. This trajectory reflects the same underlying pattern that has shaped the field’s entire history, evaluation methodology adapting to keep pace with what the systems it measures have become.

The organizations and practitioners best positioned for what comes next are the ones building evaluation practices flexible enough to keep adapting, rather than ones that treat their current methodology as a finished, permanent solution, since the systems being evaluated will keep changing, and an evaluation practice that stops evolving alongside them will, sooner or later, find itself measuring the wrong thing with confidence, precisely the failure mode a mature evaluation discipline exists to prevent.

How evaluation costs shape what gets measured in practice

Every form of evaluation, whether it’s a careful human review of sampled output, a large-scale model-graded pipeline, or a full benchmark run across a wide range of candidate models, carries a cost in time, compute, and skilled human attention, and this cost inevitably shapes what an organization chooses to measure rather than what it would ideally measure in some unconstrained world. A team with limited evaluation budget faces trade-offs, whether to invest more deeply in fewer dimensions or more broadly but shallowly across many, whether to prioritize the categories of usage that are easiest to sample and score or the categories that carry the most consequence but are harder and more expensive to evaluate well, and these trade-offs, made explicitly or by default, end up determining the actual shape of an organization’s evaluation coverage far more than any abstract methodological ideal does.

Recognizing this cost dynamic openly, rather than pretending evaluation coverage reflects a purely principled prioritization when it reflects what was affordable to measure, leads to better decisions about where to invest limited evaluation resources. An organization that explicitly maps its evaluation spending against its risk profile, prioritizing deeper, more expensive evaluation specifically for the categories of usage that carry the most consequence even when those categories are harder to measure, will end up with considerably more valuable coverage than one that defaults its evaluation effort toward whatever happens to be cheapest and easiest to automate, regardless of whether that easy-to-measure category is where the organization’s risk concentrates.

How evaluation interacts with the incentives of the people and organizations being measured

Once an evaluation metric becomes visible and consequential, whether it’s a published benchmark score that shapes public perception or an internal quality metric tied to a team’s performance review, the people and organizations being measured against it face an incentive to optimize specifically for that metric, sometimes in ways that improve the underlying quality the metric was meant to reflect and sometimes in ways that merely improve the reported number without any corresponding improvement underneath it. This dynamic, well understood across measurement disciplines far beyond AI evaluation specifically, means a metric that starts out as a reliable proxy for quality can gradually become a less reliable one precisely because of the attention and optimization pressure its visibility and consequence generates.

Guarding against this drift requires treating evaluation metrics themselves as things that need periodic scrutiny and renewal, not just the systems they measure, watching specifically for cases where a metric keeps improving while other, harder-to-game signals of quality, direct user feedback, careful independent audit downstream outcomes, fail to move in the same encouraging direction. An organization that notices this kind of divergence and responds by revising or supplementing its evaluation metrics is protecting the integrity of its measurement practice, while one that simply continues reporting the improving metric without investigating the divergence risks building decisions on top of a number that has quietly stopped meaning what it once did.

How third-party and independent evaluation complements an organization’s internal practice

An organization evaluating its system carries an inherent, structural limitation regardless of how rigorous its internal methodology happens to be: it has a stake in the outcome, and even the most well-intentioned internal evaluation practice can be shaped, consciously or not, by incentives to find and report favorable results. Independent, third-party evaluation, conducted by an external party without that same direct stake in the outcome, provides a different and valuable check, one considerably less susceptible to the kind of subtle, unconscious bias that can creep into evaluation conducted entirely in-house, however carefully that in-house practice is designed.

This is why mature evaluation practice increasingly includes external components alongside internal measurement, independent audits, third-party benchmark participation, external red-teaming conducted by people with no stake in a favorable outcome, each providing a form of validation an organization’s internal evaluation, however well-run, cannot fully provide on its own. An organization that relies purely on internal evaluation, however extensive, is missing this important external check, and the organizations building the most credible, trustworthy evaluation practices treat independent, third-party validation as a complement to their internal work rather than as an optional extra reserved only for the most heavily regulated or highest-stakes deployments.

How regulatory and legal expectations are beginning to shape evaluation practice

As AI systems take on more consequential roles, regulators and lawmakers in a growing number of jurisdictions have started to impose formal expectations around how those systems get evaluated before and after deployment, ranging from disclosure requirements about what evaluation was performed to substantive requirements that certain categories of system undergo kinds of assessment before they can be deployed into particular high-stakes contexts at all. This shift means evaluation is increasingly not just a matter of internal quality assurance an organization chooses to invest in at its discretion, but in a growing number of contexts, a legal or regulatory obligation with consequences for getting it wrong or skipping it entirely.

This regulatory dimension adds a practical consideration to how an organization should think about its evaluation practice: building evaluation infrastructure and documentation robust enough to satisfy not just internal quality goals but also plausible future external scrutiny, since regulatory expectations in this space are still actively evolving and an organization that only evaluates to the minimum bar currently required risks finding that bar has moved considerably by the time external scrutiny arrives. Organizations building durable evaluation practices tend to over-invest somewhat relative to today’s minimum regulatory bar precisely because they anticipate that bar rising, and documentation and rigor built in anticipation of that rise costs considerably less to build proactively than to retrofit reactively once a formal requirement or an incident forces the issue.

Common mistakes organizations make around AI evaluation

Several patterns recur often enough across organizations building AI evaluation practices that naming them directly, explicitly is worth doing before they lead to a consequential gap in understanding how a deployed system is truly performing in practice.

1. Treating any single evaluation practice, a benchmark score, a pre-launch test suite, as sufficient entirely on its own rather than one necessary piece within a considerably broader, multi-part discipline.

2. Conflating testing, monitoring, and evaluation as though they answered the exact same question, missing the distinct gap each one is specifically meant to close.

3. Investing evaluation effort heavily at one stage of a system’s lifecycle, pre-launch or post-launch, while leaving the other stages with considerably inadequate coverage.

4. Relying entirely, exclusively on automated or model-based grading without any ongoing human calibration validating that those methods still track human judgment.

5. Treating a standardized, general-purpose benchmark’s results as though they directly predicted performance on an organization’s particular use case.

6. Building custom, task-specific evaluation while entirely ignoring the broader field’s established standardized methods, losing the comparability those methods provide.

7. Producing evaluation reports with no organizational ownership or authority, letting serious findings sit unacted upon in a routine report nobody specifically owns.

8. Treating today’s evaluation methodology as a finished, permanently, entirely settled science rather than an actively, continuously evolving discipline with unresolved open problems.

9. Measuring whatever happens to be easy to measure rather than building evaluation backward from the decisions it specifically needs to inform.

10. Failing to communicate evaluation findings transparently to the actual people who need to trust the system’s results, treating evaluation as purely an internal exercise only.

11. Assuming a benchmark score, a test pass, or an agent evaluation result from an earlier model generation remains equally meaningful once considerably newer, more capable models are involved.

12. Neglecting the safety, reliability, and alignment dimensions of evaluation entirely while over-indexing purely, heavily on raw capability measurement alone.

13. Allowing evaluation authority to remain purely advisory in high-stakes deployments, rather than giving evaluation findings weight to delay or block a decision.

14. Treating evaluation as a purely technical exercise entirely disconnected from the organizational and governance structure needed to act on its findings.

15. Underinvesting in evaluation entirely during early development, treating it as a late-stage formality rather than a discipline that should shape decisions from the earliest stages onward.

16. Defaulting evaluation spending toward whatever happens to be cheapest and easiest to automate, rather than explicitly mapping investment against the organization’s actual risk profile.

17. Continuing to report an improving evaluation metric without ever investigating whether harder-to-game signals of underlying quality have moved alongside it too.

18. Relying purely on internal, self-conducted evaluation without any independent, third-party validation to check for the subtle, unconscious bias that inevitably shapes even well-intentioned in-house measurement.

19. Treating today’s minimum regulatory evaluation bar as a permanent, fixed target rather than anticipating that external scrutiny will likely demand considerably more in the near future.

What connects all nineteen of these mistakes is a single underlying pattern: treating AI evaluation as a narrow, one-time technical task rather than the broad, continuously evolving discipline it is, spanning multiple distinct sub-practices, multiple different dimensions of quality, every stage of a system’s actual lifecycle, and a growing web of external expectations around it, none of which can substitute for the others and all of which require sustained organizational investment rather than a single technical solution applied once and then left entirely unattended thereafter.

The deeper principle underneath all of this is that knowing whether an AI system works is not a question with a single, final answer that gets settled once and then holds true forever, it’s an ongoing demanding project that has to keep pace with a system that itself keeps changing, with usage that keeps evolving, with external scrutiny that keeps rising, and with a broader field whose understanding of how to measure these systems well is still actively, continuously maturing, and only an organization that treats evaluation with the exact same seriousness and sustained investment it gives to building the system in the first place can claim to know, with well-founded confidence, whether that system truly does what it’s supposed to do.