# Benchmark A standardized test or task suite used to measure and compare AI model performance. Common benchmarks include MMLU (Massive Multitask Language Understanding), HumanEval (code generation), GSM8K (math reasoning), ARC (science reasoning), and HellaSwag (common-sense reasoning). Model releases are accompanied by benchmark scores; leaderboards rank models by benchmark performance; and investors, journalists, and policymakers use benchmark scores as proxies for model capability. ## The Goodhart's Law problem "When a measure becomes a target, it ceases to be a good measure." AI benchmarks are the clearest contemporary illustration of Goodhart's Law. Labs optimize specifically for benchmark performance — training on benchmark-similar data, fine-tuning for benchmark formats, and selecting architectures that score well on published tests. A model that scores 90% on MMLU may not be 90% as capable as a human on the same topics — it may simply be good at the specific question format MMLU uses. The result: benchmark scores have become increasingly unreliable indicators of real-world capability. A model that tops the leaderboard on standard benchmarks may perform poorly on novel, real-world tasks. Conversely, a model that scores lower on benchmarks may be more useful in practice because its training prioritized real-world helpfulness over test-taking. ## The rhetorical function Benchmarks serve AI labs' marketing needs: a new model launch is accompanied by a chart showing benchmark improvements over competitors. But the choice of benchmarks is itself strategic — labs highlight benchmarks where their model leads and downplay benchmarks where it doesn't. The audience (investors, journalists, policymakers) rarely has the technical literacy to evaluate whether the highlighted benchmarks measure anything meaningful. "We beat GPT-4 on 7 out of 10 benchmarks" is the AI industry's equivalent of "4 out of 5 dentists recommend" — technically true, strategically selected, and potentially misleading. ## See Also - [[Frontier Model]] — the systems benchmarks attempt to rank - [[Scaling Laws]] — benchmark scores as evidence for scaling trends - [[Red-Teaming]] — the adversarial alternative to static benchmarks