What is agent evaluation?

Summary slides

Why agent evaluation is distinct from testing a system before it ships

Testing an AI system, in the sense of building a test suite that a change either passes or fails before it ships, answers a narrow question: does this particular version of the system behave correctly against this particular set of known scenarios. That question matters enormously, but it’s a fundamentally different question from the one agent evaluation exists to answer, which is broader and considerably more open-ended: how is this system performing against the unpredictable, constantly shifting stream of usage it’s facing right now, today, in production. A system can pass every single test in its pre-launch suite and still perform considerably poorly once deployed, because the test suite, however comprehensive it happens to be, was built well in advance against scenarios someone specifically anticipated, while usage inevitably includes an entire range of scenarios nobody anticipated at all beforehand.

This is precisely why evaluation has to be a continuous ongoing practice rather than a single discrete gate a system passes through once and then leaves behind for good. A test suite is binary by design, a version either passes or it doesn’t, but evaluation is fundamentally about tracking a moving target, quality as it exists in the world right now, which shifts as usage patterns shift, as the underlying model gets updated, as new tools get added, and as the population of users interacting with the system changes over time in ways no static test suite was ever built to capture.

What dimensions agent evaluation needs to cover for a deployed agent

A comprehensive agent evaluation practice tracks several distinct dimensions simultaneously, and treating any one of them as a stand-in for overall quality produces a badly incomplete picture. Task success asks whether the agent is accomplishing what users are asking it to do, the most obvious dimension but also, on its own, an insufficient one. Efficiency asks how much it costs, in time, in compute, in the number of steps taken, to reach that success, since an agent that eventually succeeds after an excessive number of retries and tool calls is meaningfully worse than one that succeeds directly, even when both are counted as successes by a pure task-completion metric.

Safety and constraint adherence ask whether the agent stayed within its intended boundaries along the way, not just whether it reached the goal, a dimension that becomes considerably more important as agents are given broader permissions to take consequential action. User satisfaction, gathered through explicit feedback, implicit signals like whether a user abandoned an interaction partway through, or whether they had to rephrase or repeat a request, captures something none of the other dimensions directly measure: whether the actual person on the other end of the interaction came away with what they needed. A team that tracks only one or two of these distinct dimensions, however carefully and thoroughly, is effectively optimizing against an incomplete, partial picture of what truly matters for a deployed agent’s overall quality in practice.

How evaluation samples production traffic rather than a fixed test set

The single most important structural difference between evaluation and testing is what each one is run against. A test suite runs against a fixed, curated set of scenarios, chosen in advance by whoever built the suite. Evaluation, by contrast, has to be run against a representative sample of production traffic, the actual requests users are sending, because that traffic is the only source of ground truth for how the system is performing right now, and no advance curation, however careful, can substitute for observing what usage looks like.

Sampling this production traffic well is itself a non-trivial design problem in its own right. A naive random sample can systematically under-represent the rare but consequential interactions that matter most a complex multi-step task that only occurs in a small fraction of usage but carries disproportionate risk when it goes wrong deserves more evaluation attention than its raw frequency alone would suggest, which is why mature evaluation practices often use stratified sampling, deliberately over-sampling from categories of interaction, high-stakes requests, unusually long conversations, interactions that ended in an explicit complaint, that carry more evaluative signal per sampled instance than an equivalent number of routine, low-stakes interactions would.

How to build rubrics for evaluating an agent’s open-ended, multi-step behavior

Evaluating a single factual answer against a known correct response is comparatively straightforward, but evaluating an agent’s extended, multi-step behavior toward an open-ended goal requires a considerably more structured rubric, since there’s rarely a single correct trajectory and a reviewer, whether human or automated, needs clear criteria to judge consistently rather than relying on a vague, holistic sense of whether the interaction went well. A well-built rubric breaks an interaction down into checkable dimensions, did the agent correctly understand what was being asked, did it choose an appropriate approach given the tools available, did it recover sensibly from any errors it encountered along the way, did it clearly communicate what it did and why, rather than asking a reviewer to render one holistic judgment that inevitably conflates several separate questions into a single number.

Building a well-constructed rubric like this requires deep domain expertise about what constitutes good behavior for the kind of agent being evaluated, since the criteria that matter for a customer-support agent are different from the criteria that matter for a coding agent or a research agent, and a generic, one-size-fits-all rubric applied uniformly across different agent types tends to miss the failure modes that matter most for each one. The most useful rubrics are developed iteratively over real time, starting from an initial, rough draft, then refined repeatedly against examples where reviewers disagreed, using that disagreement as a signal that the rubric’s criteria weren’t enough to produce consistent, reliable judgment across different reviewers.

How human review fits into an ongoing evaluation practice at scale

Human review remains the most reliable single source of evaluative judgment, a careful, well-briefed reviewer applying a well-built rubric will catch nuance that automated methods routinely miss, but it doesn’t scale to the volume of interactions a production agent generates, which means human review has to be deployed deliberately and selectively rather than applied uniformly across everything. The most effective use of human review time targets it at the interactions where it adds the most value: a stratified sample covering the full range of interaction types, a focused sample of interactions flagged by automated methods as potentially problematic, and a periodic calibration sample used specifically to check whether the automated evaluation methods running at larger scale are still tracking human judgment accurately.

This calibration role is easy to underweight but is nonetheless essential to the whole practice, since automated evaluation methods, covered in more detail below, can drift away from what a careful human reviewer would conclude without that drift ever becoming visible unless someone is periodically checking. A human review practice that only ever looks at flagged, potentially problematic interactions, without any ongoing calibration sample checking the automated methods against human judgment, risks a slow, undetected divergence where the automated scores keep reporting healthy numbers while quality, as an actual person would judge it, has quietly moved somewhere else entirely.

How automated, model-graded evaluation scales beyond what human review alone can cover

Using a separate model to grade an agent’s interactions against a defined rubric is what makes evaluation tractable at the volume a production agent generates, since it can be applied to every single interaction, or a very large sample of them, at a cost and speed no human review practice could ever match. This scaling comes with the exact same risk that applies to any model-graded evaluation broadly: the grading model’s judgment carries its own biases and blind spots, and a rubric that looks precise on paper can still be applied inconsistently by an automated grader in ways that only become visible through the kind of human calibration sampling described above.

The most reliable automated evaluation setups break a rubric into narrow sub-questions rather than asking a grading model to render one holistic score, since a narrow question, did the agent’s final action match what the user explicitly asked for, is considerably easier for a model to answer consistently than a broad, holistic question like was this interaction good overall, which invites the same kind of inconsistent, hard-to-calibrate judgment from an automated grader that it would from an under-specified human rubric. Decomposing evaluation into narrow, well-defined sub-questions, then combining those individual sub-scores according to a clear, explicit method, produces evaluation results that are both more consistent and considerably, meaningfully easier to debug whenever something looks wrong.

How evaluation tracks trends over time rather than single point-in-time scores

A single evaluation score, measured once, tells a team where the system stands at that moment, but the more valuable signal usually comes from watching how that score moves over time, since a gradual decline that would be invisible in any single snapshot becomes immediately obvious once plotted as a trend, and a sudden discontinuity, a sharp drop coinciding with a deployment or an underlying model update, points directly at a likely cause in a way an isolated score never could. This is why mature evaluation practices are built around dashboards tracking key metrics continuously, rather than periodic one-off evaluation reports produced occasionally and then set aside until the next review cycle.

Trend tracking also matters because it changes what counts as a meaningful signal versus ordinary noise. A single interaction scoring poorly is expected and unremarkable, some fraction of any large volume of interactions will always score poorly for reasons that don’t reflect any underlying systemic problem, but a metric that has been trending steadily downward across several consecutive weeks is a different, more serious signal that deserves investigation, and a team that only checks evaluation scores sporadically, without a continuous trend view, will struggle to distinguish between these two very different situations until a slow decline has already become a serious, entrenched problem.

How evaluation should be tied to user segments and use cases

A blended, system-wide evaluation score can look healthy while masking serious problems concentrated in a segment of usage, a particular user population, a particular category of request, a particular integration point, and a team that only ever tracks the blended number will miss exactly the kind of concentrated, segment-problem that matters most for the actual people affected by it. Breaking evaluation results down by segment, by use case, by user type, by request category, surfaces this kind of concentrated problem in a way an aggregate score never can, since a segment representing a small fraction of overall volume can still be performing considerably worse than the system’s average without moving that blended average enough to be noticeable.

This segment-level view matters most for exactly the categories of usage that carry the most consequence, since a segment that’s small in volume but high in stakes, a category of request tied to financial decisions, medical guidance, legal consequences, deserves dedicated evaluation attention even when its raw contribution to overall traffic volume would never earn it that same attention if evaluation resources were allocated purely proportionally to volume.

How evaluation connects specifically to hallucination evaluation and benchmarking as complementary practices

Agent evaluation, as covered here, focused on an already-deployed system’s ongoing real-world performance, sits alongside rather than replaces the more specialized practices of hallucination evaluation, which drills specifically into factual reliability with its dedicated claim-verification tooling, and agent benchmarking, which serves a different, earlier purpose entirely, comparing candidate models and architectures against each other before a deployment decision gets made in the first place. Each of these three practices answers a different question, and none of them substitutes for the others.

A team that relies purely on benchmark results to judge an already-deployed agent’s ongoing quality is making the same mistake as relying on a pre-launch test suite to judge post-launch performance, since a benchmark measures a candidate’s general capability under controlled conditions before deployment, not how that deployed system is behaving against the traffic it’s facing right now. Similarly, a team that folds hallucination checking entirely into a generic evaluation rubric without any dedicated claim-level verification will systematically under-detect the confident fabrications that a broader, more holistic rubric was never built to catch with the precision that dedicated hallucination evaluation provides. The three practices work best as a coordinated whole, benchmarking informing what to build on, evaluation tracking how the deployed result is performing, and hallucination evaluation drilling into one high-stakes dimension of that broader evaluation with considerably more depth than a general rubric alone could provide.

How to build feedback loops from evaluation findings back into actual system improvement

Evaluation that never changes anything is, in a practical sense, wasted effort, however rigorous its methodology, and the difference between an evaluation practice that improves a deployed system and one that merely produces reports nobody acts on comes down to whether there’s a working feedback loop connecting evaluation findings back to actual engineering and product decisions. This means evaluation findings need to be specific and actionable rather than a single vague aggregate score, a finding that the agent frequently fails on a particular category of multi-step request, with concrete examples attached, gives an engineering team something they can investigate and fix, while a finding that overall quality is at seventy-eight percent gives them nothing actionable to work from.

The most effective evaluation practices build this feedback loop explicitly into their process, routing categorized findings directly to the teams that own the relevant part of the system, tracking whether a finding led to a change, and then re-evaluating specifically to confirm whether that change improved the metric it was meant to address. Without this explicit closing of the loop, evaluation risks becoming a ritual that produces numbers without ever connecting those numbers to the decisions that would improve the system those numbers are meant to describe.

How evaluation should treat an agent’s efficiency and cost, not just its output quality

An agent that produces excellent output quality while consuming an excessive number of tool calls, an excessive amount of computation, or an excessive amount of real time to get there is not delivering the same value as one that reaches equivalent quality more efficiently, and an evaluation practice that tracks quality alone, without any accompanying view of cost and latency, will systematically miss this important trade-off. This matters considerably more once an agent is operating at production scale, where a marginal quality improvement purchased at a disproportionate increase in cost or latency may not be worth the trade for a deployment, even though a quality-only evaluation would report it as a straightforward improvement.

Tracking efficiency alongside quality also surfaces a distinct category of problem that a pure quality metric would never reveal on its own: an agent that’s technically still succeeding at its assigned tasks but has quietly become considerably less efficient over time, taking more steps, more retries, more tool calls to reach the same outcomes it used to reach more directly, a degradation that only becomes visible once cost and step count are tracked as their dedicated metrics rather than folded silently into an aggregate quality score that never surfaces them.

How evaluation accounts for an agent’s safety and constraint-following behavior in live deployment

Safety evaluation in a live, deployed setting carries stakes that pre-launch safety testing, however thorough, can never fully anticipate, since users interact with a deployed agent in ways no pre-launch red-teaming exercise fully predicted, and an ongoing evaluation practice needs its dedicated tracking of constraint violations, boundary tests, and near-miss incidents that occurred in actual usage rather than relying entirely on the safety assurance a system earned before it ever shipped. This means specifically monitoring for cases where an agent took an action outside its intended scope, however rare, and treating each such case not as an isolated incident to quietly note but as an evaluation finding deserving the same kind of investigation and root-cause analysis as any other significant quality problem.

This ongoing safety tracking matters because an agent’s real-world boundary-testing behavior, whether users deliberately or accidentally push it toward its edges, evolves over time as usage patterns shift and as people become more familiar with what an agent can and can’t do, which means a safety profile that looked solid at launch can degrade over time in ways only continuous, ongoing evaluation would catch, rather than something a one-time pre-launch safety review could ever be expected to guarantee indefinitely.

How evaluation criteria themselves need to evolve as a deployed agent’s usage patterns shift

The rubrics, sampling strategy, and metrics an evaluation practice was originally built around reflect an understanding of usage as it existed at the time that evaluation practice was designed, and usage doesn’t stay fixed, new use cases emerge, existing use cases shift in their relative volume, and the failure modes that mattered most at launch are frequently not the same ones that matter most a year later once a system has been tested against a much broader range of usage than it initially launched with. An evaluation practice that never revisits its rubrics and sampling strategy risks measuring against a picture of usage that’s increasingly out of date, continuing to report reassuring scores against criteria that no longer reflect where the current risk lives.

This means evaluation itself needs periodic review, not just the system it’s evaluating, checking whether the rubric’s criteria still reflect what matters for current usage, whether the sampling strategy still covers the categories of interaction that carry the most weight today, and whether new categories of usage that didn’t exist when the evaluation practice was first built have since emerged and now deserve their dedicated evaluation attention. Treating an evaluation practice as a fixed, finished artifact, rather than something that itself needs ongoing maintenance, is one of the more subtle ways a well-built evaluation system can quietly stop reflecting the reality it was built to track.

How to build organizational processes and ownership around ongoing evaluation

Evaluation that exists purely as a technical pipeline, without clear organizational ownership of what happens with its results, tends to atrophy into a system that produces numbers nobody is specifically responsible for acting on, which is why the most effective evaluation practices pair their technical infrastructure with an explicit process: a defined cadence for reviewing evaluation trends, a clear owner responsible for triaging findings and routing them to the right team, and a defined escalation path for findings that indicate a serious, urgent problem rather than routine, expected variation.

This organizational layer matters as much as the technical evaluation pipeline itself, since a team with excellent evaluation infrastructure but no clear process for acting on what it surfaces will, in practice, behave identically to a team with no evaluation practice at all, the moment findings stop translating into actual decisions and changes. Building this ownership explicitly, rather than assuming it will emerge naturally once the technical infrastructure exists, is part of what separates an evaluation practice that improves a deployed system over time from one that merely documents its problems without ever addressing them.

How evaluation should treat regressions introduced by an underlying model update

An agent’s underlying model rarely stays fixed indefinitely, providers ship new versions, teams upgrade to newer releases chasing better raw capability, and each such change is an opportunity for a deployed agent’s real-world behavior to shift in ways that weren’t part of any deliberate, intentional change to the agent’s logic, prompts, or tools. This creates an evaluation challenge distinct from the kind of gradual drift covered elsewhere: a model update can produce an abrupt, discontinuous shift in evaluation metrics that has nothing to do with anything the team building the agent changed, and distinguishing this kind of externally introduced regression from an internally caused one matters enormously for knowing where to direct an investigation once a metric moves.

The most reliable way to catch this category of regression is running evaluation, particularly the automated, model-graded layer covering broad volume, specifically bracketed around the timing of any underlying model change, comparing evaluation results from immediately before and immediately after the switch on an otherwise identical sample of interactions, rather than only ever looking at evaluation trends in the aggregate where a model-driven regression can blend into ordinary week-to-week variation and go unnoticed for considerably longer than it should. A team that upgrades an underlying model without this bracketed before-and-after comparison is effectively deploying an unevaluated change and hoping the agent’s broader evaluation trends will eventually reveal any problem, a considerably slower and less reliable way to catch exactly the kind of regression a deliberate, targeted comparison would surface immediately.

How evaluation should handle the challenge of measuring long-running, asynchronous agent tasks

Many of the evaluation techniques described so far assume an interaction resolves within a reasonably short window, long enough to sample, review, and score without much delay between the interaction happening and its evaluation being complete, but a growing share of agent deployments involve tasks that run considerably longer, an agent working through a multi-hour project, checking back periodically, handing off partial results before eventually completing, and this kind of long-running, asynchronous interaction breaks several of the assumptions a shorter-interaction evaluation pipeline was originally built around. A task that hasn’t finished yet can’t be fully scored against a completion-based rubric, and a task that spans hours or days accumulates enough intermediate state that scoring only its final result misses whatever happened, well or badly, throughout the considerable time in between.

Evaluating this kind of long-running task well means building rubrics that can meaningfully score partial progress, not just a completed final state, checking whether an agent maintained a coherent plan throughout an extended task even before that task finished, and specifically tracking whether the agent correctly recognized and adjusted for its earlier mistakes across the task’s extended duration, rather than only ever checking its behavior at the single terminal moment a shorter-task evaluation pipeline is built to expect. A team whose evaluation practice was designed entirely around short, quickly-resolving interactions and never explicitly extended to handle these considerably longer-running tasks will find its evaluation coverage quietly thinning out precisely in the category of usage, extended, high-effort, multi-step work, that often carries the most consequence when it goes wrong.

How evaluation results should be communicated to people outside the team that produced them

An evaluation practice can be technically rigorous, well-sampled, carefully calibrated against human judgment, and still fail to influence decisions if the way its results get communicated to the people who need to act on them doesn’t match how those people think about the system. An engineering team deep in the evaluation pipeline’s internals can slip naturally into reporting results in terms that make perfect sense to them, a sub-metric’s score, a particular rubric dimension’s percentage, but that mean very little to a product leader or an executive stakeholder trying to understand whether the deployed agent is getting better or worse and whether a change is worth making. Communicating evaluation results well means translating them into the terms the actual audience for that communication cares about user impact business consequence risk exposure, rather than the internal vocabulary the evaluation pipeline itself was built around.

This translation work matters because evaluation findings that never reach the people with the actual authority to act on them, in a form those people can use, are functionally equivalent to findings that were never produced at all. A well-run evaluation practice invests deliberate effort into this communication layer, building dashboards and summaries specifically tailored to different audiences rather than assuming a single technical report will serve everyone’s actual needs equally well, and treats getting a finding into the hands of the right decision-maker, in language that decision-maker can act on, as just as much a part of the evaluation practice’s job as the underlying measurement work that produced the finding in the first place.

Common mistakes teams make around agent evaluation

Several patterns recur often enough across teams running agent evaluation that naming them directly, explicitly is worth doing before they lead to a quality problem going unnoticed for far, considerably longer than it should have been allowed to.

1. Treating a pre-launch test suite as sufficient assurance entirely on its own, without building any dedicated ongoing evaluation practice for after the system goes live.

2. Reporting a single blended quality score without ever breaking it down by user segment, use case, or the category of request being evaluated.

3. Sampling production traffic purely, entirely at random, missing the rare but consequential interaction types that deserve deliberately over-sampled evaluation attention.

4. Asking human or automated reviewers to render one holistic quality judgment rather than decomposing evaluation into narrow, consistently answerable sub-questions.

5. Using automated, model-graded evaluation at scale without any ongoing human calibration sample checking whether that automated grading still tracks human judgment.

6. Reviewing evaluation results only periodically rather than tracking them continuously, missing gradual trends that a single point-in-time snapshot could never reveal at all.

7. Treating benchmark results as an adequate substitute for evaluating how an already-deployed system is performing against its live traffic.

8. Folding hallucination checking into a generic quality rubric rather than running dedicated, claim-level hallucination evaluation directly alongside broader quality tracking.

9. Producing evaluation reports with no explicit feedback loop connecting findings back to the actual engineering or product decisions that would address them.

10. Tracking output quality alone without any accompanying view of cost, latency, or step count, missing when an agent quietly becomes less efficient over real time.

11. Treating pre-launch safety testing as sufficient assurance indefinitely, without any ongoing monitoring for constraint violations that only ever emerge in live usage.

12. Leaving evaluation rubrics and sampling strategy fixed indefinitely, never revisiting them as usage patterns and emerging use cases shift over time.

13. Building excellent evaluation infrastructure without any clear organizational ownership of triaging and acting on what that infrastructure surfaces.

14. Allocating evaluation attention purely proportional to raw traffic volume, under-covering small but high-stakes segments that carry disproportionate consequence.

15. Confusing a system’s rare individual poor-scoring interaction, fully expected in any large volume of usage, with a systemic trend that deserves investigation.

16. Upgrading an underlying model without running a bracketed before-and-after evaluation comparison, letting a model-driven regression blend invisibly into ordinary week-to-week variation.

17. Building evaluation rubrics and pipelines entirely around short, quickly-resolving interactions, leaving long-running, asynchronous tasks with thinning, inadequate evaluation coverage.

18. Reporting evaluation results purely in internal technical vocabulary, missing the translation work needed for those findings to reach and influence the people with authority to act on them.

What connects all eighteen of these mistakes is a single underlying pattern: treating evaluation as a one-time gate rather than a continuous, living practice that has to track a moving target usage that shifts, drifts, and evolves in ways no static pre-launch check or one-time benchmark could ever fully anticipate in advance. An agent that was thoroughly tested and strongly benchmarked before launch can still degrade, drift, or develop entirely new failure modes well after it ships, and only an evaluation practice built to run continuously, rather than once, has any meaningful chance of catching that degradation before the actual people relying on the system do instead.

The deeper principle underneath all of this is that a deployed AI agent’s quality isn’t a fixed property established once at launch and then simply assumed to persist indefinitely, it’s a dynamic, ongoing outcome shaped continuously by usage, by underlying model changes, by shifting user populations, and by the slow, steady accumulation of edge cases no pre-launch process, however thorough, could ever fully enumerate in advance, and only a team that treats evaluation as an ongoing, structural, load-bearing part of how a deployed system is run, rather than a box checked once before shipping and never revisited again, can keep pace with how that quality continuously changes over the system’s entire live lifetime.