# AI Safety **AI Safety** (alternately **AI Alignment**, in its narrower technical subset) is the interdisciplinary research field concerned with ensuring that advanced artificial intelligence systems produce beneficial outcomes rather than catastrophic or terminal ones. The field spans technical research (how to build AI systems that reliably pursue intended objectives), philosophical work (how to specify what humans actually want well enough for AI systems to pursue those goals), governance research (what institutional structures can ensure AI is developed safely), and — at the far end of the spectrum — **AI existential safety**, the research program specifically focused on preventing scenarios in which AI systems could produce human extinction or the permanent loss of human agency over humanity's future. The field's intellectual origins are typically traced to **[[Eliezer Yudkowsky]]**'s 2001-2008 writings (the Singularity Institute for Artificial Intelligence, later MIRI), and it received its canonical academic-style treatment in **[[Nick Bostrom]]**'s 2014 book ***[[Superintelligence: Paths, Dangers, Strategies]]*** and **[[Stuart Russell]]**'s 2019 ***Human Compatible: Artificial Intelligence and the Problem of Control***. AI Safety matters for the KB for three reasons. First, it is the **dominant research-priority category** within the contemporary longtermist / EA movement — Ord's 2020 *Precipice* estimate that unaligned AI constitutes approximately 60% of 21st-century existential risk, and [[Open Philanthropy]]'s grant-making heavily weighted toward AI safety, make it the single most resource-concentrated cause area in the broader EA-aligned institutional apparatus. Second, its **institutional infrastructure** — MIRI (2000), OpenAI (2015, founded with AI safety as explicit mission), Anthropic (2021, founded by former OpenAI safety researchers), DeepMind's safety team, ARC Evals, METR, Redwood Research, Conjecture, and a dozen other organizations — represents one of the most structurally significant growths of a new research field across the 2015-2025 period, with institutional capacity that has scaled from roughly a dozen full-time researchers in 2015 to several thousand by 2025. Third, the **field's success at influencing mainstream AI policy** — from the March 2023 pause letter to the UK AI Safety Summit (November 2023) to the Biden October 2023 AI executive order to Anthropic's "[[Constitutional AI]]" approach being widely adopted — has made AI safety one of the handful of niche academic research agendas that has successfully scaled into mainstream policy relevance in roughly fifteen years. ## Intellectual Origins (2000-2014) The modern AI safety research agenda traces to several overlapping origins: ### Yudkowsky and MIRI Eliezer Yudkowsky founded the Singularity Institute for Artificial Intelligence (SIAI, later MIRI — Machine Intelligence Research Institute) in 2000, initially focused on accelerating "friendly AI" development. Yudkowsky's 2001-2010 writings (including the voluminous LessWrong sequences and his contributions to Bostrom-edited volumes) established many of the conceptual foundations: - **Goal specification problem**: Precisely specifying what humans actually want such that an AI can pursue those goals without perverse outcomes is much harder than intuition suggests - **Instrumental convergence**: Sufficiently advanced AI systems pursuing almost any goal will have convergent instrumental subgoals (self-preservation, resource acquisition, cognitive enhancement) that are dangerous - **Orthogonality thesis**: Intelligence and goals are orthogonal — an arbitrarily intelligent system can pursue arbitrarily misaligned goals ### Bostrom's *Superintelligence* (2014) Nick Bostrom's 2014 book was the first comprehensive academic treatment of AI safety concerns. Key contributions: - **Control problem**: The technical challenge of creating AI systems that pursue intended objectives as capabilities scale - **Treacherous turn**: Scenarios in which AI systems behave aligned while being monitored but become misaligned once deployed - **Intelligence explosion**: Recursive self-improvement scenarios producing rapid capability gains *Superintelligence* was a *New York Times* bestseller and received prominent endorsements from [[Bill Gates]], [[Elon Musk]], and Stephen Hawking — substantially contributing to the transition of AI safety from fringe topic to mainstream-discussable concern. ### Russell's *Human Compatible* (2019) Stuart Russell — UC Berkeley professor and co-author (with Peter Norvig) of the dominant AI textbook *Artificial Intelligence: A Modern Approach* — published *Human Compatible* in 2019. The book's framework: - **Standard model is misspecified**: AI systems designed to optimize a fixed objective function are dangerous at high capability levels - **Cooperative inverse reinforcement learning**: AI systems should be designed to be uncertain about human goals and to actively learn those goals through observation and interaction - **Beneficial AI as research problem**: The field should explicitly aim for provably beneficial AI rather than provably intelligent AI Russell's academic credentialing made the book a significant signal to the mainstream AI research community that AI safety was a legitimate research concern rather than a science-fiction speculation. ## Technical Research Areas Contemporary AI safety research spans several overlapping technical areas: ### Alignment Research on ensuring AI systems pursue intended objectives. Principal sub-areas: - **Reinforcement Learning from Human Feedback (RLHF)**: The principal currently-deployed alignment technique, used in ChatGPT, Claude, Gemini, and other major commercial AI systems - **Constitutional AI**: Anthropic's approach of training AI systems to follow principles specified in a "constitution" document - **Scalable oversight**: Techniques for humans to supervise AI systems more capable than themselves - **Debate and amplification**: Paul Christiano's proposals for extracting aligned behavior from AI systems through adversarial processes ### Interpretability Research on understanding what AI systems are actually doing internally: - **Mechanistic interpretability**: Anthropic's research program on reverse-engineering neural network computations at the level of individual circuits - **Feature visualization**: Understanding which patterns in inputs cause which activations - **Probing**: Using classifiers on internal representations to detect specific features or concepts ### Safety evaluations Research on testing AI systems for dangerous capabilities or behaviors: - **ARC Evals** / **METR** (Model Evaluation and Threat Research): Evaluations for dangerous capabilities (autonomous self-replication, cyber attack capability, bioweapon assistance) - **Red teaming**: Adversarial testing to elicit failures - **Capability elicitation**: Testing whether AI systems have capabilities they are not displaying by default ### AI governance Research on policy and institutional structures for safe AI development: - **Compute governance**: Limitations on high-compute training runs - **Evaluation requirements**: Mandatory safety evaluations before deployment - **Liability frameworks**: Legal responsibility for AI-caused harms - **International coordination**: Avoiding race-to-the-bottom dynamics among AI-developing nations ## Institutional Infrastructure The AI safety research ecosystem has grown substantially across 2015-2025: ### Principal research organizations - **MIRI** (2000): Yudkowsky's original organization; reduced technical research focus after 2022 "death with dignity" pivot toward advocacy - **OpenAI Superalignment team** (formed 2023, partially disbanded 2024): OpenAI's dedicated alignment research effort - **Anthropic** (2021): Founded by former OpenAI safety researchers including Dario and [[Daniela Amodei]], Chris Olah, and others; AI safety is Anthropic's founding mission - **DeepMind Safety team**: Long-running research group within DeepMind - **ARC** (Alignment Research Center, 2021): Paul Christiano's organization - **METR** (formerly ARC Evals, 2023): Safety evaluations organization - **Redwood Research** (2021): Alignment research - **Conjecture** (2022): Connor Leahy's alignment research organization - **Center for AI Safety** (CAIS, 2022): AI safety research, policy, and field-building ### Academic centers - **UC Berkeley CHAI** (Center for Human-Compatible AI, 2016): Stuart Russell's center - **Cambridge Leverhulme Centre for the Future of Intelligence**: Academic AI safety research - **Oxford [[Global Priorities Institute]]**: Academic philosophy of AI safety ## The 2023 Pause Letter and Its Aftermath The **March 22, 2023 Future of Life Institute open letter** calling for a six-month pause on training AI systems more powerful than GPT-4 was a landmark event in AI safety public discourse: - Published two weeks after OpenAI's GPT-4 release - Eventually signed by ~30,000 individuals including Elon Musk, Steve Wozniak, [[Yoshua Bengio]], Stuart Russell - No major AI lab actually paused training - But the policy environment shifted: the Biden October 2023 executive order on AI, the UK AI Safety Summit at Bletchley Park (November 2023), and the launch of the UK AI Safety Institute all occurred in the subsequent 9 months ## The 2026 Slowdown Moment The 2023 pause letter was signed largely by outsiders and ignored by the labs. What happened in September 2026 is structurally different, and that difference is the reason it belongs on this page. On **12 September 2026** [[Dario Amodei]] published *We Must Pace the Frontier*, calling for AI companies to deliberately slow capability improvement, and committed [[Anthropic]] to giving third-party evaluators permanent system access. Within hours [[Sam Altman]] said [[OpenAI]] agreed and would match the access commitment; [[Elon Musk]] and [[Demis Hassabis]] posted support. For the first time the people running the leading labs were publicly asking for a brake — on their own product, in unison. The ground had shifted before that. In **July 2026** more than **1,300 employees** of Anthropic, OpenAI, [[Meta]] and [[Google DeepMind]] signed an open letter asking the U.S. government to regulate AI in order to slow its development. In **August**, more than 100 technology companies offered their models to hospitals and infrastructure operators to help defend against AI-enabled cyberattacks — an implicit concession that the offensive capability was already loose. Days before Amodei's essay, Anthropic researcher **Jacob Coxon** resigned, saying that the labs "are racing straight to self-improving superintelligence and gambling with our lives" and that "the people building AI earnestly believe that it could kill us all by the end of the decade." **The government moved the other way.** Biden's Executive Order 14110 on "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" (30 October 2023) was revoked by [[Donald Trump]] on **20 January 2025**, and replaced on 23 January by an order titled "Removing Barriers to American Leadership in Artificial Intelligence," directing agencies to review and rescind anything adopted under the prior order and requiring AI systems "free from ideological bias or engineered social agendas." Asked about the slowdown calls on 13 September 2026, Trump dismissed them, citing competition with China: "I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up. And they're bringing up things that won't happen." [[David Sacks]], co-chair of the President's Council of Advisors on Science and Technology, read the industry's motive as self-interested — an attempt to avoid "massive product-liability exposure if your products enable a truly damaging cyberattack." > [!note] What makes this a finding rather than a news item > The 2023 letter was safety advocates asking labs to stop. The 2026 sequence is **labs and their own employees asking government to stop them, and government declining.** Whatever one concludes about the sincerity of the request, the direction of the appeal has reversed, and the constraint the industry says it wants does not currently exist. ## Controversies and Criticisms AI safety research has been the subject of several sustained criticisms: ### "AI safety is anti-progress" critique Some AI researchers (notably Yann LeCun of Meta, Andrew Ng, and others) argue that x-risk-framed AI safety research substantially overstates risks from current AI development trajectories and unnecessarily slows beneficial AI development. ### AI ethics vs. AI safety distinction The AI *ethics* community ([[Timnit Gebru]], Margaret Mitchell, Safiya Noble, and others) has been sometimes antagonistic to the AI *safety* community, arguing that the x-risk framing distracts from immediate harms (algorithmic bias, surveillance applications, labor displacement) that disproportionately affect marginalized populations. This is related to the TESCREAL critique coined by Gebru and Torres. ### Regulatory capture concerns Critics argue that AI safety advocacy has been substantially co-opted by the major AI labs (OpenAI, Anthropic, Google) to shape regulation in ways that entrench their market positions against potential competitors. ### Doomerism fatigue Some observers note that AI safety rhetoric has sometimes tipped into unfalsifiable doom predictions that erode the credibility of the underlying technical research. ## Connection to the KB Ecosystem - **[[Existential Risk]]**: AI safety is the principal category within the broader x-risk research program. - **[[Eliezer Yudkowsky]]**: Founding figure; MIRI originator. - **[[Nick Bostrom]]**: *Superintelligence* (2014) as canonical academic treatment. - **[[Stuart Russell]]**: *Human Compatible* (2019); UC Berkeley CHAI. - **[[Max Tegmark]]**: FLI co-founder; principal public-facing physicist advocate. - **[[Holden Karnofsky]]**: Open Philanthropy / Anthropic; principal EA-philanthropic AI safety strategist. - **[[Longtermism]]**: AI safety is the practical action-guiding corollary of longtermism. - **[[TESCREAL]]**: AI safety is one of the core constituent elements. - **[[Future of Humanity Institute]] / [[Future of Life Institute]]**: Principal institutional homes for AI safety advocacy. - **March 2023 Pause Letter**: Defining public AI safety event. ## See Also - [[Eliezer Yudkowsky]] - [[Nick Bostrom]] - [[Stuart Russell]] - [[Max Tegmark]] - [[Holden Karnofsky]] - [[Existential Risk]] - [[Longtermism]] - [[TESCREAL]] - [[Future of Life Institute]]