# Existential Risk **Existential risk** — often abbreviated as **x-risk** — is the conceptual category that organizes the contemporary longtermist and EA-adjacent research program concerned with possible catastrophic or terminal threats to humanity's long-term future. The canonical definition, from [[Nick Bostrom]]'s 2002 essay *Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards* (published in the *Journal of Evolution and Technology*), is that an existential risk is **"one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development."** The definition is deliberately broader than extinction — it includes scenarios in which humanity survives but its long-term potential is irrecoverably damaged (e.g., "lock-in" to a permanent dystopia, or the permanent loss of the capacity for technological and moral development). [[Toby Ord]]'s 2020 book *The Precipice* is the most widely-read contemporary systematic treatment of the existential risk research program, providing numerical estimates for various x-risk sources across the 21st century. Existential risk matters for the KB for three reasons. First, it is the **conceptual grounding for the entire longtermist research program** — the claim that "we are in a uniquely pivotal period with respect to existential risks" is the load-bearing empirical claim that converts longtermism from an abstract moral position (future people matter equally) into an action-guiding one (AI safety, biosecurity, and other x-risk reduction work are the most important causes). Without the empirical claim about present-period x-risks, longtermism produces no specific action prescriptions. Second, it is one of the **central intellectual categories** in the TESCREAL cluster of thinking that has substantially shaped Silicon Valley's self-understanding in the 2015-2025 period — existential risk discourse has moved from fringe academic speculation in 2002 to mainstream policy discussion (UK [[AI Safety]] Summit 2023, Biden AI executive order, UN discussions) in roughly two decades, largely through the operational work of FHI, FLI, [[Open Philanthropy]], and their allies. Third, the **specific rankings and estimates** of which x-risks are most pressing (AI-caused, biosecurity, nuclear, great power conflict, climate-related, asteroid, supervolcano, natural pandemic) have substantial practical consequences for grant-making allocations, policy advocacy priorities, and research funding flows — the longtermist movement's heavy weighting of AI risk (Ord estimates AI as contributing ~60% of total 21st-century x-risk probability) is what produces its concentration of resources on AI safety rather than on other potential catastrophic risks. ## The Bostrom Definition (2002) Bostrom's 2002 essay introduced the term "existential risk" as a replacement for the looser "global catastrophic risk" to capture a specific threshold: - **Scope**: Affects humanity as a whole, not just subsets - **Severity**: Terminates humanity or its long-term potential - **Irreversibility**: Unlike most catastrophes which allow recovery, x-risks preclude the possibility of recovery Bostrom's original taxonomy distinguished: - **Bangs**: Rapid terminal events (asteroid impact, engineered pandemic, hostile superintelligent AI, nuclear war) - **Crunches**: Slow decline scenarios in which humanity persists but loses capacity for technological or civilizational recovery - **Shrieks**: Permanent locks into undesirable end-states (permanent totalitarianism, permanent stagnation) - **Whimpers**: Slow erosion of humanity's long-term potential through value drift or mediocre trajectories The taxonomy was subsequently refined in Bostrom's 2013 paper *Existential Risk Prevention as Global Priority* and further by Ord's 2020 *Precipice* treatment. ## *The Precipice* (2020) — Ord's Systematic Treatment Toby Ord's 2020 book-length treatment provides the most widely-cited numerical estimates. Ord's headline estimate is that the probability of existential catastrophe in the 21st century is approximately **1 in 6** (~17%), with the risk dominated by anthropogenic sources: ### Ord's estimates of 21st-century x-risk sources: - **Unaligned artificial intelligence**: ~1 in 10 (~60% of total x-risk) - **Engineered pandemics**: ~1 in 30 - **Other anthropogenic risks** (including unforeseen and "other" scenarios): ~1 in 30 - **Nuclear war**: ~1 in 1,000 - **Climate change**: ~1 in 1,000 (direct existential contribution; tail risks higher) - **Natural pandemics**: ~1 in 10,000 - **Asteroid / comet impact**: ~1 in 1,000,000 - **Supervolcanic eruption**: ~1 in 10,000 The numerical estimates are explicitly rough and Ord presents them with substantial uncertainty, but the **qualitative conclusion** — that anthropogenic and particularly AI-related risks dominate total 21st-century x-risk — has been widely influential and is the empirical basis for the longtermist movement's concentration of resources on AI safety. ## The Research Program Existential risk research has developed as a distinct interdisciplinary field since Bostrom's 2002 essay. Principal institutional homes: - **Future of Humanity Institute** (Oxford, 2005-2024): Principal academic research center under Bostrom's direction until its April 2024 closure - **Centre for the Study of Existential Risk** (Cambridge, 2012-present): Cambridge-based center founded by Huw Price, Martin Rees, and [[Jaan Tallinn]] - **Future of Life Institute** (2014-present): Research funding and public advocacy organization - **Global Catastrophic Risk Institute** (2011-present): U.S.-based research organization - **Open Philanthropy Longtermism team**: Principal grant-making arm for x-risk research - **AI alignment research labs** (MIRI, ARC, etc.): Focused on the AI-specific x-risk The research program includes both substantive risk analysis (estimating probabilities, identifying intervention points) and meta-level work (moral philosophy of risk aggregation, epistemology of low-probability high-consequence reasoning, decision theory under deep uncertainty). ## Controversies and Criticisms Existential risk discourse has been the subject of several sustained criticisms: ### The "pascal's mugging" objection If the value at stake in an x-risk is astronomical (the entire future of intelligent life), then even tiny probabilities of x-risk produce expected-value calculations that dominate any other consideration. Critics argue this makes the framework structurally unable to distinguish between well-evidenced and speculative x-risks. ### Opportunity cost and present suffering Critics from within the broader EA movement (particularly the global-health wing) and outside it argue that x-risk prioritization diverts resources from well-evidenced near-term interventions (global health, direct cash transfers, animal welfare) toward speculative long-term risks with unclear intervention pathways. ### The [[Émile Torres]] / TESCREAL critique Philosopher Émile P. Torres has been one of the most sustained external critics of the existential risk / longtermist research program. The [[TESCREAL]] framework (coined by Torres and [[Timnit Gebru]]) frames the x-risk / longtermist movement as an ideological cluster with problematic genealogy (eugenics, transhumanism) and problematic present-day effects (concentration of resources in Silicon Valley, tech-billionaire political influence). ### AI-risk skepticism A significant subset of AI researchers and policy figures (including Yann LeCun of Meta, Andrew Ng, and others) dispute that current AI development trajectories produce meaningful existential risk, arguing that the FHI / FLI / Bostrom framework overstates AI capability timelines and understates the difficulty of the scenarios in which AI could produce terminal outcomes. ### Political coordination failures Some political analysts argue that the existential risk framework, while conceptually useful, has not produced effective political coordination — the concrete actions (AI pause letter, Biden executive order, UK AI Safety Summit) have had limited effect on actual AI development pace, and the movement's concentration of influence in Silicon Valley venture capital has produced concerning governance outcomes. ## Existential Risk in Policy Discourse (2015-2025) The x-risk framework has had substantial policy uptake across the 2015-2025 period: - **March 2023 FLI pause letter**: Explicit invocation of x-risk framing - **Biden October 2023 AI executive order**: Includes AI safety provisions that substantially reflect x-risk research community concerns - **UK AI Safety Summit (November 2023)**: Bletchley Park summit convened by Rishi Sunak; x-risk discourse was central - **UN AI High-Level Advisory Body**: Includes members with x-risk research backgrounds - **Biosecurity policy**: BWC review conference discussions have substantially integrated x-risk framing for engineered pandemic scenarios - **Nuclear policy**: Less pronounced uptake, though x-risk framing has entered some nuclear policy discussions ## Connection to the KB Ecosystem - **[[Nick Bostrom]]**: Canonical definition; principal intellectual progenitor. - **[[Toby Ord]]**: Author of the most-cited systematic treatment (*The Precipice*, 2020). - **[[William MacAskill]]**: Longtermism co-founder; *What We Owe the Future* elaborates the x-risk grounding. - **[[Anders Sandberg]]**: FHI senior researcher with substantive x-risk technical contributions. - **[[Max Tegmark]]**: FLI co-founder; principal public advocate for x-risk framework. - **[[Eliezer Yudkowsky]]**: Principal advocate for the AI-specific x-risk case, with more aggressive urgency than the Bostrom/Ord mainstream. - **[[Future of Humanity Institute]]**: Principal academic research center through April 2024. - **[[Future of Life Institute]]**: Principal research-funding and advocacy organization. - **[[Longtermism]]**: X-risk is the empirical grounding concept for longtermism's practical prescriptions. - **[[TESCREAL]]**: X-risk is one of the constituent elements of the TESCREAL cluster as Torres/Gebru diagnose it. - **[[Effective Altruism]]**: Longtermist wing of the EA movement is substantially x-risk-focused. ## See Also - [[Nick Bostrom]] - [[Toby Ord]] - [[Anders Sandberg]] - [[Max Tegmark]] - [[Eliezer Yudkowsky]] - [[Future of Humanity Institute]] - [[Future of Life Institute]] - [[Longtermism]] - [[TESCREAL]] - [[Effective Altruism]]