# Eval / Evaluation Systematically testing what an AI model can and can't do — and, increasingly, what it will and won't do. An eval might test coding ability (can the model solve these programming problems?), factual accuracy (does it get these questions right?), safety properties (does it refuse dangerous requests?), or behavioral tendencies (is it sycophantic? biased? manipulative?). ## Why "eval" became its own vocabulary The abbreviated form "eval" (always lowercase, often pluralized as "evals") has become standard industry jargon, signaling insider status the way "repo" does in software development. The term carries weight because evals are where competing claims about AI capabilities are tested — and often deflated. The political economy of evals is significant. Labs choose which evals to publish (the ones their model does well on) and which to omit. [[Benchmark]] are standardized evals, but labs increasingly create custom evals designed to showcase their model's specific strengths. The result is an evaluation ecosystem that looks rigorous but is often strategically curated. The governance dimension: as AI regulation develops, "evals" are becoming the mechanism through which compliance is assessed. The EU AI Act, executive orders on AI safety, and voluntary commitments from labs all reference evaluation processes. Who designs the evals, who runs them, and who interprets the results are governance questions with real policy consequences. Third-party evaluation organizations (METR, Apollo Research, UK AISI) exist specifically to provide independent assessment, but the eval ecosystem remains largely self-regulated. The internal culture: at frontier labs, "running evals" is a significant portion of model development. OpenAI's "preparedness" team, Anthropic's RSP (Responsible Scaling Policy) evaluations, and [[Google DeepMind]]'s safety evaluations all represent institutional commitment to evaluation — though critics argue these are necessary but insufficient, given that labs evaluate their own products. ## See Also - [[Benchmark]] — standardized evals - [[Red-Teaming]] — adversarial evaluation - [[Model Card]] — where eval results are reported - [[Reward Hacking]] — why evals are unreliable measures of true capability - [[Responsible AI]] — the governance framework evals serve