What are LLM benchmarks?

Summary slides

Why LLM benchmarks exist and what question they answer

Before a model is ever wired into an agent, given tools, or deployed into a product, there’s a more basic question worth answering on its own: how capable is this model at the kinds of tasks language models are generally asked to do, answering factual questions, solving math problems, writing and debugging code, understanding and generating natural language across a range of registers and domains. LLM benchmarks exist specifically to answer this question, isolating a model’s raw generative capability from any particular application it might eventually be built into, which is precisely what makes them useful as an early filtering signal long before anyone has decided what product that model will power.

This isolation is also what limits an LLM benchmark’s usefulness beyond that early filtering role. A model’s score on a benchmark measuring its ability to answer static, well-defined questions says relatively little about how it will behave once it has to call tools, once it has to maintain a plan across many steps, or once it has to recover gracefully from an unexpected error mid-task, which is exactly the gap that agent benchmarks were built to address separately. Understanding an LLM benchmark’s actual scope, what capability it measures and what it deliberately leaves out, is the necessary starting point for using its results well rather than reading more into a reported score than that score was ever designed to support.

What separates an LLM benchmark from an agent benchmark

An LLM benchmark typically presents a model with a single, self-contained prompt, a question, a problem, a passage to summarize, and evaluates the single response that comes back, with no environment for the model to act inside, no tools to call, and no multi-step trajectory to observe. This structural simplicity is a strength, since it makes LLM benchmarks considerably cheaper to run at scale and considerably easier to score consistently than an agent benchmark’s multi-step trajectories, but it also means an LLM benchmark simply cannot measure the planning, tool use, and error-recovery behavior that only shows up once a model is embedded in a longer, interactive loop.

This is why the two kinds of benchmark answer different questions rather than one being a more basic version of the other. A model can score exceptionally well on LLM benchmarks measuring its raw knowledge and reasoning while performing considerably worse as an agent, because agent performance depends on additional capabilities, planning across steps, using tool outputs correctly, recognizing and recovering from its errors, that a single-turn benchmark never exercises. A team evaluating a model purely for its raw generative quality, writing assistance, question answering without external tools, will find LLM benchmarks directly relevant, while a team evaluating a model specifically for an agentic deployment should treat strong LLM benchmark scores as a necessary but insufficient signal on their own.

What the major categories of LLM benchmarks measure

Knowledge benchmarks test whether a model can correctly recall and apply factual information across a broad range of domains, history, science, law, medicine, typically through large collections of multiple-choice questions spanning many subject areas, and a strong score here reflects how much accurate factual information a model absorbed during training and how reliably it can retrieve that information when asked directly. Reasoning benchmarks test something meaningfully different, whether a model can work through a multi-step logical or mathematical argument correctly, and a model can score well on knowledge benchmarks while performing considerably worse on reasoning benchmarks if it has broad factual recall but struggles to chain that knowledge together through the kind of careful, sequential logical steps a reasoning problem demands.

Coding benchmarks test whether a model can produce working, correct code against a specification, typically verified by running the generated code against a set of test cases rather than comparing it textually against some reference solution, since two functionally identical solutions can look completely different as text while being equally correct. Language understanding benchmarks test more subtle capabilities, whether a model correctly interprets ambiguous phrasing, tracks entities and their relationships across a long passage, or produces text that’s fluent and appropriate for a register and audience, capabilities that don’t reduce as cleanly to a single right-or-wrong answer as factual recall does and that consequently require more sophisticated scoring approaches than a simple correctness check.

How multiple-choice benchmarks work and what their scoring means

A multiple-choice benchmark presents a model with a question and several candidate answers, and a model’s response gets scored by checking whether it selected the answer matching the known correct one, a format that scales extremely well, since scoring is fully automatic and completely unambiguous, no human judgment or model-based grading required at all. This scalability is precisely why so many widely cited LLM benchmarks are built in this multiple-choice format, spanning tens of thousands of questions across dozens of subject areas that would be prohibitively expensive to score through any method requiring human review or careful automated grading of open-ended text.

The format’s simplicity comes with a limitation worth understanding clearly: a model can sometimes select the correct answer among multiple choices through a kind of pattern-matching or elimination that doesn’t reflect an understanding of the underlying material, recognizing the correct answer’s phrasing as more plausible-sounding than the others without necessarily being able to independently derive or explain that same answer if asked to produce it as open-ended text rather than simply selecting it from a list. This is part of why a model’s multiple-choice benchmark score, however impressive, doesn’t automatically predict equally strong performance on open-ended tasks that require generating a correct answer from scratch rather than merely recognizing one among a small set of provided options.

How open-ended generation benchmarks are scored differently

A benchmark that asks a model to write an essay, generate a summary, or produce free-form code can’t be scored through a simple exact match against one single correct answer, since there are often many different valid ways to accomplish the same underlying task correctly, which means open-ended benchmarks need considerably more sophisticated scoring approaches than their multiple-choice counterparts. For tasks with a checkable functional outcome, generated code that either passes a test suite or doesn’t, scoring can remain fully automatic and objective even though the format is open-ended, since the verification happens against the actual behavior of the output rather than against its literal text.

For open-ended tasks without any single checkable functional outcome, evaluating the quality of a written summary or the appropriateness of a piece of creative writing, scoring typically relies either on human judgment against a defined rubric or on a separate model prompted to grade the response, both approaches carrying their trade-offs, human judgment is more reliable but far more expensive and slower to run at scale, while model-based grading scales considerably better but inherits whatever blind spots and biases the grading model itself carries. Understanding which scoring method an open-ended benchmark uses matters for interpreting its results correctly, since a benchmark scored by an imperfect automated grader carries a distinct source of noise that a benchmark scored by careful, calibrated human review does not.

How benchmarks measure reasoning ability specifically

Reasoning benchmarks are typically built around problems that can’t be solved through simple factual recall alone, math word problems requiring several sequential steps, logic puzzles requiring careful, structured deduction, multi-step inference problems where the correct answer only becomes apparent after correctly working through several intermediate conclusions in the right order. What makes these benchmarks informative is that they can distinguish between a model that has memorized a problem’s answer from its training data and one that’s working through the underlying logical structure, since a well-constructed reasoning benchmark includes enough variation in the numbers and framing of otherwise structurally similar problems that pure memorization stops being a reliable strategy for scoring well.

A particularly informative technique in reasoning evaluation checks not just whether a model’s final answer is correct but whether the intermediate reasoning steps that led to that answer are sound, since a model can occasionally arrive at a correct final answer through flawed or coincidentally lucky reasoning, a distinction a benchmark that only checks the final answer will completely miss. Benchmarks that specifically evaluate a model’s step-by-step reasoning trace, rather than just its final output, provide a considerably more reliable signal of reasoning capability, since they catch cases where a model’s apparent success masks an underlying process that would fail to generalize reliably to a problem framed even slightly differently.

How code-generation benchmarks work and what they verify

A code-generation benchmark presents a model with a natural-language description of a programming task and evaluates the code it produces by executing that code against a predefined set of test cases, checking whether the generated solution produces the correct output across a range of inputs rather than comparing the generated code textually against any single reference implementation. This execution-based verification is what makes code benchmarks unusually rigorous compared to many other LLM benchmark categories, since there’s no ambiguity in what counts as success, the code either produces correct results across the test cases or it doesn’t, without any of the interpretive judgment that scoring open-ended natural-language output requires.

This rigor comes with its limitation worth understanding: a code benchmark’s test cases can never be fully exhaustive, and a generated solution that passes every provided test case can still contain a bug that simply wasn’t covered by those tests, which means a high score on a code-generation benchmark reflects strong performance against that benchmark’s particular test coverage rather than a guarantee of fully correct code in every conceivable situation. The most carefully constructed code benchmarks specifically include edge cases, unusual inputs, boundary conditions, error-handling scenarios, precisely to catch generated solutions that handle the obvious common cases correctly while failing on the less obvious ones a less careful test suite would never have caught in the first place.

How benchmark contamination happens and how it’s mitigated

A benchmark’s questions and their correct answers eventually appear across the public internet, in academic papers discussing the benchmark, in blog posts analyzing model performance, sometimes in the benchmark’s repository if access isn’t carefully controlled, and once that happens, a model trained on internet data collected after that point may have effectively memorized the benchmark’s answers rather than demonstrating the underlying capability the benchmark was built to measure. This is called contamination, and it’s one of the most persistent, difficult challenges in benchmark design, since a contaminated benchmark can continue producing impressive-looking scores for models that would perform considerably worse on novel problems of comparable underlying difficulty.

Benchmark maintainers fight contamination through several distinct strategies: keeping a portion of questions permanently private and never publishing them, periodically releasing entirely new versions of a benchmark once the previous version’s answers have become too widely available, and specifically testing whether a model’s performance correlates suspiciously closely with the exact wording of publicly available solutions rather than with equivalent problems phrased differently. A reader relying on a benchmark’s reported scores should specifically check whether that benchmark has any documented contamination mitigation, and should treat scores with skepticism whenever a model’s training data plausibly overlaps with a benchmark that has been publicly discussed for a considerable amount of time.

Why benchmark saturation happens over time and what it means

A benchmark that was difficult when first introduced can become considerably easier within a relatively short time as model capability improves broadly across the field, and once every leading model under comparison scores close to the maximum possible score, that benchmark has effectively reached saturation, no longer able to meaningfully distinguish between a strong model and a merely adequate one, even though it continues to be cited in comparisons. This pattern has repeated many times across the history of language model evaluation, a benchmark that once represented a frontier of difficulty eventually becoming a baseline every serious model is simply expected to clear, prompting researchers to build new, more difficult benchmarks that restore the ability to discriminate between models of meaningfully different capability.

Saturation matters for how a reader should interpret benchmark results, since a saturated benchmark’s continued high scores tell you considerably less about a model’s actual relative capability than they might initially appear to, precisely because every strong model is clustering near the ceiling regardless of underlying differences in capability. A reader evaluating recent models should specifically check whether a benchmark still shows meaningful score variation among leading models, and should treat a benchmark where every top model scores within a percentage point or two of each other as one that’s stopped providing much useful discriminating signal, however historically significant that benchmark once was.

Why aggregate leaderboard rankings can be misleading

A leaderboard combining scores across several different benchmarks into one single aggregate ranking invites a reader to treat that ranking as a straightforward, single measure of overall model quality, but the weighting a leaderboard applies across its constituent benchmarks, how much a coding benchmark counts relative to a knowledge benchmark relative to a reasoning benchmark, is itself an editorial choice, and a different, equally reasonable weighting scheme can produce a different overall ranking from the exact same underlying set of individual scores. A model ranked first on one leaderboard’s particular weighting can rank considerably lower on another leaderboard that weights the same constituent benchmarks differently, without either ranking being objectively more correct than the other.

The most useful way to read a leaderboard involves looking past the single aggregate number entirely and examining the individual benchmark scores that make it up, since a model’s strengths and weaknesses are considerably more visible in that individual breakdown than in any single blended ranking, and a team choosing a model for a particular use case should weight the individual benchmarks relevant to that use case according to its actual priorities rather than deferring to whatever aggregate weighting a particular leaderboard happened to choose for its general-purpose ranking.

How benchmarks handle multilingual and cross-lingual capability

Most widely cited LLM benchmarks were originally built in English, reflecting where the bulk of early benchmark development happened, and a model’s strong score on an English-language benchmark says relatively little about its actual capability in other languages, since training data volume, quality, and representation vary enormously across languages, and a model that performs excellently in English can perform considerably worse in a language with less training data available, even when the underlying task and difficulty level are held constant. This gap matters enormously for any deployment intended to serve a multilingual user base, where an English-only benchmark score provides essentially no direct evidence about the actual quality a non-English-speaking user will experience.

Dedicated multilingual benchmarks have been built specifically to address this gap, testing the same underlying capabilities, knowledge, reasoning, translation quality, across a deliberately broad range of languages rather than assuming English performance generalizes. A team evaluating a model for deployment in a language or region should specifically seek out benchmark results in that actual language rather than inferring quality from an English-only score, since the gap between English and non-English performance can be considerably larger than intuition, shaped mostly by exposure to English-dominated benchmark reporting, would tend to suggest.

How safety and alignment benchmarks differ from capability benchmarks

Capability benchmarks measure what a model can do, whether it can correctly answer a question, solve a problem, generate working code, while safety and alignment benchmarks measure something meaningfully different, whether a model behaves appropriately when asked to do something it shouldn’t, correctly refusing harmful requests, avoiding producing dangerous or misleading content, behaving consistently and predictably rather than being easily manipulated into abandoning its intended behavior through clever adversarial prompting. A model can score exceptionally well on capability benchmarks while performing considerably worse on safety benchmarks, and the reverse is equally true, an overly cautious model can score well on safety benchmarks while unnecessarily refusing a considerable number of entirely legitimate requests that a less cautious model would have handled correctly.

This is why a comprehensive evaluation of a model considers capability and safety benchmarks as separate, complementary dimensions rather than treating a single capability score as an adequate proxy for overall model quality, since neither dimension can substitute for the other and a team deploying a model into any context involving users needs visibility into both. Safety benchmarks themselves carry their methodological challenges, since the line between an appropriate refusal and an unnecessarily overcautious one is itself debatable and depends heavily on the context a model is being deployed into, which is part of why safety benchmark design remains a considerably less mature, more actively contested area than the design of purely capability-focused benchmarks.

How private, held-out evaluation sets work

Some of the most trusted benchmark results come from evaluation sets that are never published at all, held privately by the organization running the evaluation and used specifically to test models without any risk of the kind of contamination that affects fully public benchmarks. A private evaluation set trades away the transparency and independent reproducibility that a public benchmark offers, an outside reader has no way to directly inspect the questions being used, in exchange for a considerably stronger guarantee against the contamination risk that undermines confidence in public benchmark results over time as those results circulate more widely.

This trade-off means a reader encountering a reported score from a private evaluation set should weigh it differently than a score from a fully public, independently reproducible benchmark, treating the private result as informative but somewhat less independently verifiable, and should specifically look for signals of rigor, whether the organization running the private evaluation has a track record of careful, credible reporting, whether the evaluation methodology itself has been documented even if the underlying questions haven’t been, before placing as much confidence in a private score as in a fully transparent, publicly reproducible one.

How benchmark results relate to real-world usefulness

A model’s strong benchmark scores across knowledge, reasoning, and coding measure capability, but real-world usefulness depends on considerably more than raw capability alone, how well a model follows instructions, how appropriately it handles ambiguous or underspecified requests, how naturally it maintains a coherent, helpful tone across an extended interaction, dimensions that traditional benchmarks were never really built to measure with much precision at all. A model can score impressively on every major capability benchmark while still frustrating users through poor instruction-following or an unhelpful conversational style, a gap that only becomes visible once a model is deployed against usage rather than a fixed set of benchmark questions.

This is why benchmark results function best as an early filtering signal rather than a final verdict on real-world quality, narrowing a wide field of candidate models down to a smaller set worth evaluating against a deployment’s actual usage, exactly the same relationship agent benchmarks have to the system-testing and ongoing evaluation practices that ultimately determine whether a model performs well once deployed. A team that selects a model purely on the strength of its benchmark scores, without any further evaluation against its actual use case, risks discovering the gap between benchmark performance and usefulness only after users have already encountered it directly.

How benchmarks are run and why methodology details change results considerably

Running a model against a benchmark involves considerably more implementation decisions than simply feeding it questions and checking the answers, and small differences in exactly how those decisions get made can shift a reported score by a meaningful margin even when two evaluations are nominally testing the exact same model against the exact same benchmark. The prompt template used to present a question, whether it includes a handful of worked examples before the actual question, how strictly the parsing logic extracts a final answer from a model’s full response, whether that response is allowed extended reasoning before answering or forced to answer immediately, all of these implementation choices affect the resulting score, sometimes by several percentage points, independent of any actual difference in the model’s underlying capability.

This methodological sensitivity is exactly why comparing benchmark scores reported by different organizations, each running their independent implementation of the same nominal benchmark, has to be done with caution, since an apparent gap between two models on a shared benchmark can partly or entirely reflect a difference in evaluation methodology rather than a difference in capability. The most trustworthy, reliable comparisons come from a single organization running every model under comparison through the exact same evaluation pipeline with identical prompting and parsing, since that at least holds methodology constant even if it can’t eliminate every other source of variation, and a reader comparing scores pulled from different original sources should specifically check whether those sources used comparable methodology before treating the resulting gap as a meaningful signal about relative model capability.

How adversarial and dynamically generated benchmarks address the limits of static test sets

A traditional benchmark’s fixed set of questions, however large and carefully constructed, is inherently vulnerable to the saturation and contamination problems already covered, since a static, unchanging collection of questions eventually becomes both memorized and too easy as the field’s overall capability advances around it. Adversarial benchmarks take a different approach, specifically constructing questions designed to probe a model’s weaknesses rather than sampling broadly across a domain, often built by having human experts or even other models actively search for prompts that cause failures, then compiling those specifically discovered failure cases into a benchmark that concentrates its difficulty precisely where models are most likely to struggle.

Dynamically generated benchmarks go a step further, generating fresh questions programmatically according to a template rather than drawing from any fixed, pre-written collection at all, which makes memorization essentially impossible since the exact questions a model encounters during evaluation didn’t exist anywhere before that evaluation run happened. This approach trades away some of the interpretability and human-reviewed quality control that a carefully hand-curated static benchmark offers, an automatically generated question can occasionally be poorly formed or ambiguous in ways a human-reviewed benchmark would have caught, in exchange for a considerably stronger, more durable guarantee against both contamination and saturation over time, which is part of why dynamic and adversarially constructed benchmarks have become an increasingly important complement to the traditional, static benchmarks that dominated earlier evaluation practice.

How the choice of what to benchmark shapes what capabilities get prioritized

Benchmarks don’t just measure capability, they also shape what capability gets built in the first place, since a research organization deciding where to invest its limited development effort naturally gravitates toward improving whatever a widely cited benchmark happens to measure, and a capability that no prominent benchmark tracks well is considerably easier for the broader field to overlook, however useful that capability might be for deployments. This dynamic means the set of benchmarks that happen to be widely cited at any moment exerts influence over the field’s overall trajectory, and a capability gap that persists across many model generations sometimes reflects less about the underlying technical difficulty of that capability and more about the simple fact that no benchmark has made that particular gap sufficiently visible or embarrassing for anyone to prioritize closing it.

This is worth keeping in mind specifically when a team notices a mismatch between how a model performs on the widely cited benchmarks everyone discusses and how it performs on the team’s particular use case, since that mismatch isn’t necessarily a sign anything is wrong with either the model or the benchmark, it can simply reflect the fact that the team’s need happens to fall outside whatever narrower slice of capability the field’s dominant benchmarks currently happen to track closely. Building a dedicated, task-specific evaluation for a use case that existing public benchmarks don’t cover well is often a more reliable path to an informative signal than searching for some existing, widely cited benchmark that was never designed with that particular need in mind.

Common mistakes people make interpreting LLM benchmarks

Several patterns recur often enough in how people interpret LLM benchmark results that naming them directly, explicitly is worth doing before they lead to a costly model selection mistake that could have been avoided.

1. Treating a single aggregate leaderboard ranking as an objective definitive measure of overall model quality, rather than one particular editorial weighting among many equally valid, reasonable alternatives.

2. Assuming a strong English-language benchmark score predicts equally strong performance in other languages without ever checking any dedicated multilingual evaluation.

3. Relying on a benchmark’s continued high scores without ever checking whether that benchmark has become saturated and stopped meaningfully discriminating between models at all.

4. Trusting a reported score without ever seriously considering whether the underlying model’s training data could plausibly include that benchmark’s questions and answers.

5. Treating a strong multiple-choice benchmark score as evidence of equally strong open-ended generation ability, missing the gap between recognition and independent generation from scratch.

6. Selecting a model for an agentic deployment based purely on LLM benchmark scores, without any dedicated agent benchmark evaluation of its actual multi-step, tool-using behavior.

7. Treating capability benchmark scores as an adequate proxy for a model’s actual safety behavior, ignoring the distinct dimension safety benchmarks were specifically, deliberately built to measure.

8. Assuming a code benchmark’s passing score guarantees fully, entirely correct code, missing that test coverage is never exhaustive at all.

9. Comparing scores across two entirely different benchmarks as though they measured the exact same underlying capability, when their task design and scoring methodology differ considerably.

10. Placing full confidence in a model-graded open-ended benchmark score without ever checking whether that grading model’s judgments have been validated against human review.

11. Ignoring a benchmark’s reasoning-trace quality entirely, scoring only the final answer and missing cases where a correct result was reached through flawed, coincidental underlying reasoning.

12. Treating a private evaluation set’s reported score with the exact same confidence as a fully transparent, independently reproducible public benchmark’s result.

13. Choosing a model purely on the strength of its benchmark performance without any further evaluation against the actual use case it will be deployed for in practice.

14. Overlooking instruction-following and conversational quality entirely, treating raw capability benchmarks as though they fully captured real-world usefulness entirely on their own.

15. Assuming a benchmark score that was accurate for an earlier model generation remains an equally meaningful signal once considerably newer, more capable models have since been released.

16. Comparing benchmark scores pulled from different original sources without ever checking whether those sources used comparable evaluation methodology in the first place at all.

17. Treating a static, well-established benchmark as somehow immune to the contamination and saturation problems that adversarial and dynamically generated benchmarks were specifically, deliberately built to address.

18. Searching for an existing, widely cited benchmark to justify a use case rather than building a dedicated, task-specific evaluation when no existing benchmark covers that actual need.

What connects all eighteen of these mistakes is a single underlying pattern: treating a benchmark score as a complete, self-sufficient verdict rather than a narrow, carefully bounded measurement of one capability under one set of conditions, measured through one methodology that carries its limitations. A benchmark score means exactly what its underlying task design scoring methodology, and contamination safeguards allow it to mean, no more than that and every single mistake in this list traces back, in one way or another, to reading more into a reported number than the benchmark that produced it was ever built to support.

The deeper principle underneath all of this is that an LLM benchmark is a deliberately narrow, carefully bounded instrument, built specifically to isolate and measure one particular capability cleanly, and its value comes precisely, exactly from that narrowness rather than despite it, provided the reader understands exactly what narrow question a reported score is truly answering. A team that reads benchmark results with this same careful discipline, asking what precisely was measured, under what conditions, scored how, and against what honest risk of contamination, will consistently, reliably make better, more informed model selection decisions than one that simply reaches for whichever model currently tops whichever leaderboard happens to be most visible at the time, without ever looking underneath that single number.