Plate I — Why benchmarks exist
Three jobs, one tension
A benchmark is doing at least one of three jobs, and knowing which one explains most of the odd behavior around them.
Compare models fairly
Same items, same scoring rule, any model. Benchmarks are the only apples-to-apples instrument the field has — across labs, across years, across model sizes. This comparability is also why bad benchmarks persist: five years of prior results is a network effect nobody wants to abandon.
A signal to climb
Training a model means making thousands of decisions — data mixes, architectures, RL recipes — and each needs a scalar to judge it by. Internal eval suites decide which checkpoints live and which experiments die. The public benchmarks are the visible tip of this machinery.
Evidence for claims
Launches, papers, funding, procurement — numbers persuade in a way demos don’t. Which is exactly the tension: once a measure becomes a target (and a marketing asset), pressure to game it is structural, not accidental. That’s Goodhart’s law, and it shapes every benchmark’s life.
Plate II — The life of a benchmark
Every benchmark follows the same curve
Released hard, farmed for progress, then solved and abandoned — typically inside four to five years, and lately faster. Click through the stages; the worked example is GSM8K, grade-school math word problems.
Plate III — The collection
32 specimens, 8 families
The benchmarks you’ll actually meet in papers, model cards and launch posts. Filter by family or status; click any specimen for how it works, why researchers reach for it, what to distrust — and, where one exists, a link straight to its live leaderboard.
Start here — live scores across models
- LMArena ↗Live Elo from millions of blind human votes. The industry’s popularity contest.
- Epoch AI Benchmarking Hub ↗Independent reruns across GPQA, FrontierMath, SWE-bench and more — one harness for all.
- Artificial Analysis ↗Frontier models rerun independently, with cost and speed beside quality.
- HELM (Stanford CRFM) ↗Standardized, transparent evaluation across many scenarios. The academic reference.
- SWE-bench ↗The coding-agent scoreboard, with the scaffold behind each number.
- ARC Prize ↗ARC-AGI v1 and v2 standings — and cost-per-task, which headlines omit.
Plate IV — Saturation, observed
Watch six benchmarks get eaten
Best published score by year (approximate, assembled from papers and leaderboards). The pattern repeats: a new benchmark starts in the foothills, climbs for two or three years, then flattens into the shaded saturation zone — and the field moves to a harder one. Hover any tile for values.
Shaded zone ≈ saturated (90–100%). Dashed line where a human-expert baseline exists. Five of these six are now inside the band; SWE-bench Verified took three years to get there, GPQA two. Humanity’s Last Exam is the only one still climbing in open air — and it is two years old.
View the same data as a table
Plate V — Reading an eval table
The fine print is the result
A fictional leaderboard, typeset the way labs actually publish them. The superscripts and parentheticals aren’t decoration — they change what the number means. Hover the dotted terms.
| Model | GPQA-D (pass@1) | AIME ’25 (cons@64) | SWE-bench V (% resolved) | HLE (with tools) | Arena Elo |
|---|---|---|---|---|---|
| Larkspur-3 | 88.9 | 96.7 | 79.2 | 31.4 | 1451 |
| Bower-2 | 86.1 | 92.0 | 74.8 | 26.9 | 1438 |
| Petrel-40B | 68.4 | 61.3 | 41.5 | 8.2 | 1289 |
Plate VI — Field hazards
Six ways a benchmark number lies
None of these are exotic edge cases. Each one has already bent a headline result you’ve probably seen.
Contamination
Test questions leak into training data — benchmarks are published on the very internet models train on. The model isn’t solving the problem; it’s remembering the answer sheet.
Field caseGPT-4 solved 10/10 pre-2021 Codeforces problems and 0/10 published just after its training cutoff — same difficulty, different memorization.
Saturation & label noise
Near the ceiling, benchmarks stop measuring models and start measuring their own mistakes. When the top ten models sit within two points, the “gap” is mostly mislabeled questions and grading quirks.
Field caseAudits of MMLU (MMLU-Redux) found errors in a meaningful share of questions — over half in the worst subsets. The last few points aren’t winnable by being smarter.
Goodhart’s law
“When a measure becomes a target, it ceases to be a good measure.” Labs tune data mixes and RL against the public suites — scores rise faster than the general ability they were meant to proxy.
Field caseModels fine-tuned toward human-preference leaderboards drift verbose and sycophantic: what the metric rewards, the model becomes.
Judge bias
Many evals use an LLM as the grader. LLM judges prefer longer answers, confident tone, and — measurably — output that resembles their own. The judge is part of the experiment.
Field caseAlpacaEval shipped a “length-controlled” version after win rates tracked answer length. It wasn’t enough: the #1 entry on its live board today is an adversarial null model that games the judge and answers nothing.
Construct validity
Does the benchmark measure what its name implies? Multiple-choice trivia ≠ reasoning; solved GitHub issues from 12 mature Python repos ≠ your codebase. High score, narrow meaning.
Field caseModels near the top of SWE-bench Verified lose more than half their score on SWE-bench Pro’s fresher, harder repos. Same skill on paper; very different number.
Harness sensitivity
Scores are produced by a pipeline — prompt format, few-shot picks, answer parser, agent scaffold, sampling settings. Papers rarely report error bars; reruns rarely match.
Field caseIndependent reruns of published models routinely land points below the paper’s number. Not fraud — different harness. It’s why leaderboards that rerun everything themselves exist.
Plate VII — Choosing your instrument
“I want to measure…”
What a researcher would actually reach for in early 2026, by goal — with the caveat they’d mutter while doing it.