What is AI system testing?

Summary slides

Why testing an AI system differs from testing conventional software

Conventional software testing rests on a comparatively simple premise: given an input, a correctly, properly functioning system should produce a predictable output, and a test simply checks whether that expected output occurred as expected. This particular premise holds because conventional software’s actual behavior is fully determined entirely by its code, meaning the exact same input reliably produces the exact same output every single time it’s run, unless the underlying code itself has changed in the meantime. An AI system built around a language model breaks this premise in a structural way, since the same input can produce meaningfully different outputs across different runs, and even when an output differs from a previous run, that difference doesn’t necessarily mean anything has gone wrong.

This means AI system testing can’t simply ask whether an output exactly, precisely matches some single, expected answer written in advance, it has to ask a considerably, meaningfully different question instead: does this output fall within the range of outputs that would be considered correct, safe, and appropriate for this particular input. Answering that question requires testing infrastructure and testing judgment considerably, meaningfully more sophisticated than conventional exact-match testing alone, since the very notion of a test “passing” has to be redefined around a range of acceptable behavior rather than a single, fixed expected value.

How non-determinism specifically complicates the actual mechanics of testing

A conventional test suite runs once and produces a reliable, repeatable result every single time, but an AI system’s non-deterministic behavior means a single test run tells you considerably, meaningfully less than it would for conventional software, since a test might pass on one run purely by chance and fail on the next, even with absolutely nothing about the underlying system having changed between those two runs. This means AI system testing benefits from running critical tests multiple times and evaluating the actual distribution of outcomes, rather than treating a single pass or a single failure as fully conclusive on its own.

This particular repeated-run approach also considerably matters for detecting regressions, since a system that passed a test ninety-five times out of a hundred actual runs before some change was originally made, and now passes only eighty times out of a hundred after that same change, has regressed even though it’s still technically passing more often than it fails, a distinction that a testing approach built around single pass-or-fail runs would never catch. Building statistical thinking directly into how test results get interpreted, rather than simply treating every individual test run as a simple, crude binary verdict, is precisely what non-determinism requires from AI system testing that conventional software testing never had to account for in the first place.

What needs to be tested across an AI system

AI system testing really needs to cover several distinct dimensions all simultaneously, not merely whether an output is factually or functionally, technically correct in some narrow sense. Output quality asks whether a response accomplishes what the user was trying to accomplish, correctness in the conventional sense but also helpfulness, relevance, and appropriate completeness. Safety asks whether the system avoids producing harmful content and correctly refuses requests it should refuse. Robustness asks whether the system behaves consistently and appropriately across different ways of phrasing what is, at heart, the same underlying request, since a system that performs well on one phrasing but poorly on a functionally equivalent rephrasing carries practical risk once users, who phrase things in unpredictable ways start interacting with it.

Integration behavior asks whether the system correctly interacts with whatever tools data sources, and downstream systems it’s connected to in practice, a dimension that’s easy to overlook when testing focuses narrowly on the model’s raw text output in isolation, disconnected from the broader system it operates within. A model that produces excellent, correct text in isolated testing can still fail badly once integrated into a live system if that integration itself, the actual way tool calls get constructed, the way retrieved content gets incorporated wasn’t tested with the exact same rigor applied to the model’s standalone output.

How different types of testing fit together into one coherent practice

Unit-level testing for an AI system typically evaluates a narrow piece of functionality, a single prompt template, a single tool integration, entirely in isolation from the rest of the system, giving fast, focused feedback during active development before a system is fully assembled. Integration testing evaluates how these individual pieces behave once combined into the complete system, catching problems that only emerge from how components interact rather than from any single component considered entirely on its own. Regression testing specifically checks whether a change, a new model version, an updated prompt, a new tool, has degraded behavior that previously worked correctly, run specifically whenever something in the system changes rather than only at a single point before initial launch.

Red-teaming, already covered in depth elsewhere in the context of security specifically, deserves its place within a comprehensive testing practice too, in its own right, actively attempting to find inputs that produce incorrect, unsafe, or manipulated behavior, complementing the more systematic, comprehensive coverage that unit and integration testing aim to provide with the kind of creative, adversarial thinking that systematic testing alone tends to miss. Human evaluation, having actual people review a sample of the system’s outputs against judgment rather than purely automated criteria, catches subtler quality problems that automated tests, built around whatever criteria someone thought to encode in advance, simply aren’t equipped to catch on their own.

How to build a comprehensive test suite for an AI system

A test suite for an AI system needs deliberate coverage across the entire, full range of situations the system will encounter once live, not merely the easy, obviously correct cases that happen to be simplest to write tests for. This means specifically including edge cases, ambiguous requests, adversarial inputs, and unusual phrasings alongside the more straightforward, typical cases that make up the bulk of ordinary use, since a test suite dominated entirely by easy cases will systematically miss exactly the harder, more consequential failures that matter most in actual production use.

Building this coverage benefits considerably from mining production interactions, once a system has some usage history, for cases that challenged the system in ways the original test suite never anticipated, feeding those discovered cases back into the test suite so it grows more comprehensive over time rather than staying frozen at whatever coverage existed when it was first, originally written. A test suite that’s never updated after initial development inevitably falls behind the actual, evolving range of situations a system encounters once it’s live and handling unpredictable user behavior.

How to handle testing when there isn’t a single, unambiguously correct answer

Many of the actual tasks an AI system is built to handle don’t have any single, objectively correct answer at all, a good summary, a helpful response to an ambiguous question, an appropriate tone for a situation, all admit multiple acceptable answers rather than one uniquely correct one. Testing for this kind of task means defining acceptability criteria explicitly, what makes an output good enough, rather than defining a single expected output and checking for an exact match, since exact-match testing simply doesn’t apply to tasks that were never supposed to have one single correct answer in the first place.

This means AI system testing benefits considerably from rubric-based evaluation, explicit, carefully written criteria that a human reviewer, or in some cases a separate evaluation model, can apply consistently across many different outputs, checking whether each output satisfies the criteria that matter for this kind of task rather than checking whether it happens to match one single reference answer someone wrote in advance. Building well-specified rubrics takes deliberate effort upfront, but that investment pays off directly in test results that are meaningful, rather than test results that either fail correct-but-differently-phrased outputs or pass inadequate ones simply because no rubric was ever written to catch the difference.

How continuous testing works after a system is already deployed and live

Testing an AI system doesn’t stop the moment it launches, since production conditions reveal failure modes that even a thorough, careful pre-launch test suite never fully anticipated in advance, and a system’s behavior can shift after launch, through model updates, through changing content sources, through accumulated drift in stored context or memory, in ways that make continuous, ongoing testing a necessity rather than merely a nice-to-have addition to pre-launch testing. This means AI system testing needs an actual pipeline for continuously running the existing test suite against the live, current system, not merely a one-time pre-launch gate that gets checked once and then left behind.

This particular continuous testing benefits considerably from being explicitly, deliberately tied to triggers, a new model version being deployed, a meaningful change to the system’s prompts or tools specifically, rather than running purely on a fixed schedule disconnected from what’s changed in the system. A test run specifically triggered by an actual change gives an organization a direct signal about whether that change introduced a regression, while a test run on a purely fixed schedule, disconnected from actual changes, provides considerably weaker, less actionable signal about what specifically caused whatever change in test results it happens to reveal.

How AI system testing connects to broader evaluation practice and metrics

Testing and broader evaluation practice are closely related but meaningfully distinct practices: testing typically asks a binary or near-binary question, did this behavior pass or fail against defined, explicit criteria, while evaluation more broadly asks how well a system performs along some continuous dimension, tracking ongoing metrics over time rather than simply checking discrete pass or fail conditions. Testing gives confidence that known failure modes haven’t occurred; evaluation gives ongoing insight into how a system’s overall quality is trending, information testing alone, focused narrowly on pass or fail conditions, doesn’t naturally provide.

These two distinct practices work best entirely together rather than as any kind of substitute for one another, since a system that passes every individual test in its suite can still be gradually, quietly declining in overall quality along dimensions no single test was ever specifically built to catch, while a system tracked purely through aggregate evaluation metrics can still harbor a serious failure mode that only a targeted test would surface. An organization relying on only one of these two practices, testing alone or evaluation alone, is missing exactly the kind of signal the other practice was specifically built to provide.

How to design test cases that reflect user behavior

Test cases written by that exact same team that built a system tend to unconsciously reflect that team’s mental model of how the system should be used in practice, phrasing requests the way an engineer familiar with the system’s internal design would naturally phrase them, rather than the varied, sometimes ambiguous, sometimes poorly specified way users phrase requests when they’re not thinking about the system’s internal architecture at all. This particular gap between internally written test cases and real-world, actual usage is precisely the kind of blind spot that only becomes visible once a system has usage data to compare against, which is precisely why testing benefits so considerably from incorporating observed user behavior rather than relying purely on test cases the development team imagined in advance.

Actively closing this gap deliberately means actively seeking out diverse sources for test cases, involving people outside the core development team in writing or reviewing test scenarios, deliberately including phrasings and framings that feel unnatural or unlikely to the people who built the system but that actual users have been observed to use. A test suite written entirely by the system’s builders will always carry some version of this blind spot, however careful those builders try to be, simply because it’s difficult to imagine usage patterns meaningfully different from one’s accumulated intuition about how a system is supposed to be used.

How testing should specifically account for a system’s actual failure modes rather than only its successes

Much of the entirely natural, human instinct in building a test suite is to verify that a system succeeds at what it’s supposed to do in the first place, but thorough AI system testing needs equal, deliberate attention paid to how the system fails, since a system’s failure behavior matters just as much for practical safety as its success behavior does. A system that fails by clearly, visibly declining to answer at all is considerably far safer than one that fails by confidently producing a plausible-sounding but incorrect answer instead, and testing that only checks whether a system succeeds, without separately evaluating how it behaves in the cases where it doesn’t, misses this important distinction entirely.

This means testing practice considerably benefits from deliberately constructing test cases specifically designed to push a system well past its actual capability, tasks it can’t be expected to handle correctly, and then evaluating whether its failure in those cases is safe and honest, a clear, visible refusal or expression of uncertainty, rather than a confidently wrong answer that looks, on the surface, indistinguishable from a correct one. A system evaluated only on tasks well within its actual capability will never reveal this critical distinction, since it never gets the chance to fail in the first place.

How to build consensus around what “good enough” means for a system

Testing ultimately depends entirely on someone having explicitly decided, in a deliberate way, what level of performance counts as acceptable for a particular system, and this decision is harder than it first appears, since different stakeholders within the same organization often carry meaningfully different intuitions about where that bar should sit. An engineering team focused on shipping quickly and efficiently may honestly consider a level of performance acceptable that a legal or compliance team focused on risk exposure, would consider clearly, obviously inadequate instead, and without an explicit, actual process for reconciling these different perspectives, the bar a system gets tested against tends to default to whoever happens to have the most direct influence over the testing process itself, rather than reflecting a considered, deliberate organizational judgment.

Building consensus here means bringing the relevant stakeholders, not merely the engineering team building the system, into an actual, explicit conversation about what acceptable performance means for this system its actual real-world consequences, before testing criteria get finalized and locked in. A testing bar set unilaterally by whichever team happens to run the testing process, without any input from the people who understand the real-world stakes involved, risks being either needlessly conservative for a low-stakes system or dangerously permissive for a high-stakes one.

How testing costs and testing speed trade against testing thoroughness

Truly comprehensive AI system testing running many separate trials per individual test case specifically to account for non-determinism, covering a wide range of edge cases and observed usage patterns, incorporating human evaluation, carries actual cost in both compute and human time, and this cost creates a practical tension with the desire to ship changes quickly that any testing practice has to navigate rather than simply wish away. A testing regime so thorough it takes multiple days to fully complete for every single, individual change slows development down to a pace that itself becomes its problem, while a testing regime that’s fast enough to run on every single change but too thin to catch meaningful problems only ever provides the surface appearance of quality assurance without any of its actual substance behind it.

Navigating this tension well means tiering the testing thoroughness directly to the actual stakes of a change, running a fast, lightweight test suite on every single change as an immediate signal, while reserving the comprehensive, expensive testing regime for changes that are significant enough to warrant it, a new model version, a meaningful change to core prompts or tools, rather than applying the same uniform, expensive testing bar to every single change regardless of how consequential that change is. This tiering itself deserves deliberate design, since a poorly calibrated tiering scheme can just as easily under-test a significant change as it can over-test a trivial one.

How testing should account for the difference between capability and reliability

A system’s actual capability, whether it’s able to correctly handle a task at all under ideal, favorable conditions, is meaningfully distinct from its actual reliability, whether it correctly handles that same task consistently across the messy, variable conditions of actual real-world use, and testing that only ever evaluates capability under carefully controlled, favorable conditions can produce a misleading picture of how a system will perform once it’s live and facing unpredictable variation in how requests arrive. A system that succeeds at a task ninety-five percent of the time under clean, ideal, controlled test conditions might succeed considerably less often once facing the noise, ambiguity, and unexpected, unpredictable phrasing that actual users honestly introduce day to day.

This particular distinction means testing needs to deliberately introduce the kind of real-world noise and variation that capability testing entirely alone tends to exclude, testing not merely whether a system can succeed at a task under favorable conditions but whether it does succeed reliably once realistic variation, typos, ambiguous phrasing, incomplete information, unexpected formatting, gets deliberately introduced into the test cases themselves. A testing practice that only ever measures capability under ideal, favorable conditions is measuring something honestly but it’s measuring the wrong thing entirely if the actual goal is understanding how a system will perform once it’s deployed and facing the messy, unpredictable real world out there.

How testing should specifically account for the way small changes can produce disproportionate behavioral shifts

Conventional software changes tend to produce effects roughly proportionate to the actual size of the change itself, a small code modification generally produces a small, localized behavioral difference, and this rough proportionality is part of what makes conventional regression testing tractable, since a small change generally only needs to be tested against the area of behavior it touches. AI systems don’t always, reliably respect this exact same proportionality, since a seemingly minor, small adjustment to a system prompt, or a small update to an underlying model, can sometimes produce a disproportionate shift in behavior across different areas that appear, on the surface, to have absolutely nothing to do with whatever change was originally made.

This means AI system testing benefits from treating even small, seemingly minor changes with a degree of caution that conventional software testing wouldn’t normally apply to a change of comparable size, running a broad regression suite even for changes that look, at first glance, narrow and self-contained, rather than assuming a small change only warrants narrowly targeted testing focused specifically on the area it was intended to affect. An organization that calibrates its testing scope purely based on how large a change looks on paper, without accounting for this potential for disproportionate impact, risks missing exactly the kind of surprising, far-reaching regression that AI systems are specifically prone to producing.

How testing should treat the role of automated evaluation models used to grade other models’ output

A common, practical approach to scaling AI system testing uses a separate model to grade the outputs of the system currently under test, a technique that makes it possible to evaluate far more test cases than human review alone could ever realistically cover, but one that introduces its distinct risk worth naming directly: the grading model’s judgment can itself be systematically biased or simply wrong in ways that a testing practice relying entirely on it will never notice, since there’s no independent check catching the grader’s mistakes.

Mitigating this risk means periodically validating the grading model’s judgments against human review conducted independently, checking whether the automated grades correlate with what a careful human reviewer would independently conclude, rather than trusting the automated grader indefinitely once it’s initially set up and never revisiting whether its judgments still hold up over time. A grading model that quietly drifts out of alignment with human judgment over time, whether through its underlying model changing or through the test cases themselves evolving well beyond what the grader was originally calibrated against, can quietly undermine an entire testing practice built on top of it, producing results that look rigorous and comprehensive while measuring something considerably less meaningful than careful evaluation would provide.

Common mistakes teams make around AI system testing

Several patterns recur often enough across organizations building AI systems that naming them directly, explicitly is worth doing before they’re painfully discovered through an incident instead.

1. Treating a single test run as conclusive, missing that non-determinism means a reliable result only ever emerges from evaluating the distribution of outcomes across multiple, repeated runs.

2. Building a test suite dominated by easy, obviously correct cases, systematically missing the harder, considerably more consequential failures that matter most in production use.

3. Testing a model’s raw text output entirely in isolation without ever testing how it behaves once integrated into the broader system’s tools and data sources.

4. Relying purely, entirely on exact-match testing for tasks that admit multiple acceptable answers rather than one single, objectively, precisely correct one.

5. Never mining production interactions for cases that challenged the system, leaving the test suite frozen at whatever coverage existed at the system’s initial development.

6. Treating pre-launch testing as sufficient entirely on its own, without building any continuous testing pipeline for after the system is live.

7. Running continuous tests on a purely fixed, rigid schedule disconnected from actual system changes, rather than triggering them specifically by the changes whose actual impact needs to be verified.

8. Writing rubrics for subjective tasks only vaguely, loosely, leaving human or automated reviewers unable to apply consistent, meaningful criteria across different, varied outputs.

9. Treating testing and broader evaluation as entirely interchangeable, missing that one catches known failure modes while the other tracks ongoing quality trends that neither one alone fully captures.

10. Skipping red-teaming entirely in favor of purely, only systematic testing, missing the creative adversarial cases that systematic coverage alone tends not to anticipate in advance.

11. Never running human evaluation, relying entirely, exclusively on automated criteria that were only ever built to catch the particular problems someone thought to encode in advance beforehand.

12. Treating a regression as absent simply because a test still technically passes more often than it fails, missing a decline in pass rate that a purely binary view of results would never surface.

13. Building integration tests only for the happy path, never testing how the system really behaves when a tool call fails or a data source returns something unexpected instead.

14. Assuming a test suite that worked well for an earlier, considerably simpler version of a system remains adequate as that system’s actual scope and complexity considerably grow over time, well past whatever original scope that suite was designed around.

15. Treating testing as a task owned entirely, exclusively by a separate quality team disconnected from the actual engineers building and actively changing the system day to day, week after week.

16. Writing test cases that unconsciously reflect the development team’s particular mental model of the system, rather than incorporating the varied, ambiguous phrasing actual users use in practice day to day.

17. Only ever verifying that a system succeeds at its intended task, without separately evaluating whether its failure behavior is safe and honest rather than confidently, plausibly, dangerously wrong instead.

18. Letting the testing bar silently default to whichever team happens to run the testing process, without deliberate input from the stakeholders who understand the real-world stakes involved.

19. Applying a single, uniform testing thoroughness to every single change regardless of its actual stakes, either slowing development down to a crawl or providing only the surface appearance of quality assurance instead without any substance.

20. Measuring only a system’s capability under clean, ideal, favorable conditions missing how considerably, meaningfully less reliable it performs once facing real-world noise and variation.

21. Calibrating testing scope purely, simply by how large a change looks on paper, missing that even a small, seemingly minor change can produce a disproportionate shift in an AI system’s actual behavior.

22. Trusting an automated grading model indefinitely once initially set up, without periodically validating its judgments against careful human review specifically to catch drift in what it’s really measuring over time.

What connects all twenty-two of these mistakes is a single underlying pattern: applying conventional software testing assumptions, a single expected output, a one-time pre-launch gate, a purely binary pass or fail verdict, to a system whose actual behavior is non-deterministic continuous rather than binary in its quality, and capable of drifting well after initial launch in ways conventional testing was never built to catch. An AI system that passes a conventionally structured test suite can still carry serious problems that suite was never designed to detect in the first place.

The deeper principle underneath all of this is that testing an AI system means testing something whose actual correctness isn’t a fixed, binary property the way it is for conventional software, it’s a continuous question about whether a system’s actual behavior, across an enormous range of situations that can’t be fully, exhaustively enumerated in advance stays within the bounds of what’s safe, helpful, and appropriate to anyone relying on it, and an organization that tests an AI system the exact same way it would test a conventional program is measuring the wrong thing entirely, however rigorously and carefully it happens to measure it.