# Interpretability **Interpretability** (or **mechanistic interpretability**) is the effort to understand *how* AI models produce their outputs — not just what they say, but what internal computations lead to that output. Current AI models are largely "black boxes": billions of [[Parameters]] interact in ways that produce capable behavior, but researchers cannot fully explain *why* the model gives any particular answer. The stakes are high. If you deploy an AI system to analyze intelligence reports, diagnose diseases, or recommend sentences, you want to know not just that it usually gets the right answer, but *why* it reaches each conclusion — and under what conditions it might fail catastrophically. Interpretability research aims to open the black box. ## The State of the Field Interpretability research has made significant progress, particularly at Anthropic, which has made it a central research priority. Key advances include: **Feature discovery.** Researchers have identified interpretable "features" — internal representations that correspond to recognizable concepts (a feature that activates for references to the Golden Gate Bridge, a feature that tracks whether text is code vs. natural language, features that represent ethical reasoning). These discoveries show that models develop organized internal representations, not just statistical noise. **Circuit analysis.** Tracing specific behaviors to specific computational pathways ("circuits") within the model — identifying which attention heads and layers are responsible for particular capabilities. This allows researchers to understand, for instance, how the model performs arithmetic or resolves pronoun references. **Sparse autoencoders.** A technique for decomposing model activations into interpretable features, enabling systematic mapping of what the model represents internally. Anthropic's 2024 publications demonstrated this at scale on Claude. Despite this progress, interpretability remains far from complete. Current techniques can explain specific behaviors in controlled settings, but cannot provide a comprehensive account of what a large model is doing across all inputs. The gap between "we can interpret individual features" and "we understand the model" remains vast. ## Why It Matters for Governance Interpretability is the technical prerequisite for meaningful AI accountability. Without it, AI governance operates on observed behavior alone — testing what the model does, without understanding why. This is like regulating a pharmaceutical based only on clinical trials without understanding its biochemical mechanism: you can measure effects but can't predict side effects, interactions, or failure modes. The EU AI Act requires "transparency" for high-risk AI systems, but the required level of interpretability remains undefined. The gap between what regulators want (understandable AI decisions) and what the technology can currently provide (partial feature-level interpretability) is one of the central tensions in AI governance. ## KB Significance 1. **The accountability infrastructure.** Interpretability is the technical foundation for AI accountability. Without it, claims about model safety, bias, or alignment are empirical (based on observed behavior) rather than mechanistic (based on understood processes). For the KB's governance analysis, the state of interpretability determines whether AI oversight is substantive or superficial. 2. **The Anthropic differentiator.** Anthropic has invested more heavily in interpretability research than any competitor, making it central to its public identity and safety case. The KB should track whether interpretability advances translate into meaningfully safer models or remain primarily a research and marketing asset. 3. **The military accountability gap.** When AI systems make targeting recommendations or intelligence assessments in military contexts, interpretability determines whether human operators can understand and meaningfully review AI-generated conclusions — or whether "human in the loop" becomes a rubber stamp on an opaque process. ## See Also - [[Alignment]] — The problem interpretability helps solve - [[AI Safety]] — The broader field - [[Transformer Architecture]] — The architecture being interpreted - [[Parameters]] — What interpretability is trying to understand - [[Attention Mechanism]] — A component amenable to interpretation - [[Neurosymbolic AI]] — The alternative route: build systems legible by construction rather than interpreted after the fact