Plate I — Why video is different
Grading taste at scale
LLM benchmarks can check an answer. Video benchmarks must grade perception — which forces every eval into one of three shapes, and each shape fails differently.
Perception is the ground truth
A math benchmark checks the final number; “does this clip look right” has no such check. Every video eval is therefore a proxy metric (FVD), a machine judge (VBench, VQA scorers), or a human vote (arenas, panels) — three approximations of an eye, all foolable.
Models grading models
Automated judges make evaluation scalable: detectors count objects, CLIP checks alignment, aesthetic models score beauty. But a judge’s taste becomes the target — train against it and you inherit its blind spots along with its approval. The judge is part of the experiment.
The reel is a benchmark claim
Launch reels move markets more than tables do — and a reel is an eval with n = 12 cherry-picked samples. The serious instruments (arenas, fixed-prompt suites, uncurated releases like MovieGenBench) exist largely as antidotes to the demo economy.
Plate II — The subjects under study
The models these instruments measure
The frontier as of mid-2026 — and how each presents its numbers. Version numbers churn fast; the postures don’t.
Veo
Native audio + video. Markets with human side-by-sides, arena standings and filmmaker demos; a fixture near the top of preference rankings at each release.
Sora
Made “world simulator” the sales pitch. The physics benchmarks in this guide exist partly to audit exactly that claim — and it scores in the single digits on VideoPhy-2’s hard subset.
Seedance
Debuted at #1 on Artificial Analysis’ T2V and I2V arenas and has stayed there; Seedance 2.0 currently leads the with-audio board. Emphasizes multi-shot narrative.
Kling
Arena mainstay with an enormous creator base; strong image-to-video and motion controls. Launch posts lean on win-rate studies and arena Elo.
Wan
Topped VBench at release and still leads VideoPhy-2’s hard subset — an open model beating closed ones on physics. The baseline every open-video paper compares against.
LTX
Competes on latency and cost as much as quality — near-real-time generation on accessible hardware. Leads the open-weights arena board; the efficiency baseline others quote.
Plate III — The life of a benchmark
Same curve, faster clock
Video benchmarks age like LLM benchmarks — introduced hard, farmed for progress, then outgrown — but the field is younger and the churn faster. Click through the stages; the worked example is FVD, the metric that ruled video generation for five years.
Plate IV — The collection
22 specimens, 6 families
The benchmarks and metrics you’ll meet in video and image model reports. Filter by family or status; click any specimen for how it works, why labs reach for it, what to distrust — and, where one exists, a link straight to its live scoreboard.
Start here — live scores across models
- Artificial Analysis Video Arena ↗Blind human votes, Elo. Separate T2V, I2V and video-editing boards. The launch-day scoreboard.
- VBench & VBench-2.0 leaderboard ↗Dimension-level scores per model, both versions on one Space. Read dimensions, not totals.
- VideoPhy-2 leaderboard ↗Physical commonsense, open vs. closed. The hard subset is where claims go to die.
- Physics-IQ leaderboard ↗Continuation-vs-reality. Two boards — quote the stricter Verified one.
- WorldScore leaderboard ↗Camera controllability, quality and dynamics across video, 3D and 4D generators.
- MovieGenBench ↗Not a scoreboard — 1,003 prompts with uncurated outputs. Re-judge the claim yourself.
Plate V — Reading dimensions, not totals
The static-video trick
Two illustrative model profiles under a VBench-style weighted total. “Grebe 1.6” generates near-static footage: consistency and smoothness soar, motion collapses — and the headline total barely notices. This is why researchers read the dimension table, and why VBench has a dynamic-degree dimension at all. Try re-weighting.
Profiles and weights are illustrative, built to mirror a documented pattern: consistency and smoothness judges reward videos where little happens, so suite totals can favor the more static model. Real VBench totals cluster within ~2 points at the top while dynamic degree spreads by 30+.
Plate VI — Reading a launch table
The fine print is the result
A fictional launch comparison, typeset the way video labs actually publish them. Hover the dotted terms — each footnote changes what its number means.
| Model | Arena Elo | VBench (total) | Dynamic degree | VideoPhy-2 (hard, joint) | Win rate‡ |
|---|---|---|---|---|---|
| Bittern-2 | 1287 | 87.5 | 71.2 | 24.1% | 68% |
| Grebe 1.6 | 1274 | 89.1 | 43.5 | 19.8% | 61% |
| Teal-XL | 1191 | 84.0 | 77.9 | 12.3% | 44% |
Plate VII — Field hazards
Six ways a video-gen number lies
Every one of these has already bent a headline claim you’ve probably seen.
The static-video trick
Consistency and smoothness judges reward videos where nothing happens. A model can climb suite totals by moving less — beautiful stills gently drifting.
Field caseVBench added a dynamic-degree dimension specifically to catch this; read it next to consistency, never separately.
Cherry-picking
Generation is stochastic: one prompt, twenty tries, one masterpiece. Demo reels and paper figures show the best draw; you will get the median one.
Field caseMeta’s MovieGenBench released every output, uncurated, for 1,003 fixed prompts — the deliberate antidote, and still the exception rather than the rule.
Metric–eye divergence
FVD is dominated by per-frame appearance and largely blind to temporal nonsense; two videos with identical frames in scrambled order can score nearly alike.
Field caseA 2024 content-bias study showed FVD barely reacts to temporal corruptions humans spot instantly; JEDi was proposed the same year to replace its features.
Reward-hacking the judges
Aesthetic and preference models are used as RL training targets. Optimize against a taste model and you get its taste: glossy, high-contrast, centered — the “AI look.”
Field caseMuch of the visual sameness across image models traces to shared preference judges (HPS-style reward models) in their post-training loops.
Unreproducible human evals
In-house win rates ship with launch posts: internal prompts, unnamed raters, unstated sample sizes, and the rival’s version chosen by the winner.
Field caseWhen two competing launches each claim a >60% win rate over the other — and they have — at least one methodology is doing heavy lifting.
Moving task definitions
Resolution, duration, audio, shot count and hidden prompt-rewriting all change the task. Scores from different configs get quoted side by side as if comparable.
Field caseA 5-second 720p clip and a 10-second 1080p multi-shot clip with synced audio are different benchmarks wearing the same name — check the config table before the score table.
Plate VIII — Choosing your instrument
“I want to measure…”
What a practitioner would actually reach for in early 2026, by goal — with the caveat they’d mutter while doing it.