# Machine Intelligence Research Institute The **Machine Intelligence Research Institute (MIRI)** — founded in 2000 by **[[Eliezer Yudkowsky]]** and originally called the **Singularity Institute for Artificial Intelligence (SIAI)** — is the **oldest continuously-operating AI safety research organization** and one of the principal institutional origins of the contemporary AI alignment research field. Headquartered in Berkeley, California, and operating as a 501(c)(3) non-profit, MIRI's founding mission was to accelerate the development of "friendly AI" — AI systems with values aligned with human flourishing — as a strategy for ensuring that artificial general intelligence (AGI), when it emerged, would produce beneficial rather than catastrophic outcomes. Across MIRI's subsequent history, it has been one of the principal research and intellectual-community homes for the AI safety research agenda, including: foundational work by Yudkowsky and colleagues on the theoretical structure of aligned AI; the Agent Foundations research program that Nate Soares and colleagues developed; and the broader intellectual ecosystem including the LessWrong online community (launched as an outgrowth of Yudkowsky's MIRI-period writing). MIRI matters for the KB for three reasons. First, it is the **founding institution of the contemporary AI safety research field** — the organization where Yudkowsky developed the conceptual foundations (instrumental convergence, orthogonality thesis, goal specification problem, treacherous turn) that Bostrom's 2014 *Superintelligence* subsequently systematized, Russell's 2019 *Human Compatible* built on, and the entire subsequent AI alignment research community has worked within. Second, MIRI's **April 2022 "Death with Dignity" strategic pivot** — the announcement by Yudkowsky that the organization was shifting from technical alignment research toward AI safety advocacy on the grounds that technical alignment research had made insufficient progress to avert expected existential catastrophe — is one of the most documented strategic decisions in the AI safety research community, and represents the point at which MIRI's identity shifted from "research organization attempting to solve alignment" to "advocacy organization warning of expected catastrophe." Third, MIRI's **intellectual ecosystem** — including [[LessWrong]] (founded 2009 as an outgrowth of Yudkowsky's MIRI-era Sequences), the broader rationality community, the AI alignment research community, and the pipeline into the EA movement — makes it structurally central to the TESCREAL intellectual cluster that the KB documents. ## Founding (2000) as Singularity Institute Yudkowsky founded the **Singularity Institute for Artificial Intelligence (SIAI)** in 2000. The founding context: - Yudkowsky was a 20-year-old autodidact with no formal credentials but substantial writing on AI, self-improving systems, and the Singularity - His early writing had attracted some online attention, including from Brian Atkins (co-founder of Pair Networks) who provided early funding - The original SIAI mission was to *accelerate* the development of "Friendly AI" — the organization was founded on the premise that Yudkowsky or a colleague might directly build AGI, and that having AGI developed with safety considerations front-loaded was better than leaving AGI development to organizations without such concerns - This acceleration-focused framing was later revised as Yudkowsky became progressively more pessimistic about the difficulty of alignment ## The Yudkowsky Sequences and LessWrong (2006-2009) Between 2006 and 2009, Yudkowsky wrote what became known as **the Sequences** — a massive body of writing (~600,000 words) on rationality, cognitive biases, Bayesian reasoning, and AI safety. Initially published on the Overcoming Bias blog (shared with Robin Hanson), the Sequences were foundational for what became the rationality community. In 2009, the Sequences migrated to a new platform called **LessWrong**, which became the online community surrounding MIRI's intellectual work. LessWrong has been: - The principal online community for the rationality and AI safety research community - A recruiting ground for MIRI staff and for the broader AI safety research community - An intellectual incubator for concepts that subsequently became influential in the EA movement ## Research Programs MIRI's research output across its 25-year history has included: ### Friendly AI (2000s) Yudkowsky's early writing on Friendly AI established many foundational concepts: - **Coherent Extrapolated Volition (CEV)**: Yudkowsky's proposal that an aligned AI should pursue "what humans would want if they knew more, thought faster, and were more the people they wished they were" - **Goal-content integrity**: The concept that an AI system's goal preservation under self-modification is a nontrivial property requiring specific design - **Recursive self-improvement analysis**: Theoretical work on the dynamics of AI systems that improve their own capabilities ### Agent Foundations (2010s) Under Nate Soares' research leadership, MIRI developed the **Agent Foundations** research program — theoretical work on foundational questions in decision theory, logical uncertainty, embedded agency, and related formal topics: - **Logical uncertainty**: How rational agents should reason under uncertainty about mathematical facts - **Embedded agency**: How to model agents that are part of the environment they are reasoning about - **Decision theory**: Work on updateless decision theory, functional decision theory, and related formalizations - **Vingean reflection**: Formal analysis of how agents can reason about other agents more intelligent than themselves ### The 2020s Reorientation Under the changing strategic landscape of the early 2020s — particularly the rapid capability gains of GPT-3 (2020), GPT-4 (2023), and subsequent systems — MIRI's research orientation shifted. ## "Death with Dignity" (April 2022) On April 1, 2022 (despite the date being April Fools' Day, the announcement was serious), Yudkowsky published a MIRI blog post titled **"MIRI announces new 'Death with Dignity' strategy"**. The announcement: - **Thesis**: Yudkowsky's assessment was that the difficulty of the alignment problem, combined with the pace of capability gains in the AI industry, made successful alignment of the first transformative AI systems highly unlikely - **Strategic pivot**: Rather than continuing to focus primarily on alignment research, MIRI would shift toward advocacy and toward helping individual researchers and organizations navigate what Yudkowsky framed as an increasingly bleak strategic situation - **"Death with dignity"**: The framing that even in scenarios where extinction was likely, there were better and worse ways to proceed — doing alignment research, advocating for AI governance, and refusing to contribute to capability advances were framed as the ways to "die with dignity" The announcement was one of the most widely-discussed moments in the AI safety research community and substantially shaped the subsequent discourse. ## Yudkowsky's March 2023 *TIME* Op-ed In March 2023, shortly after the FLI pause letter, Yudkowsky published an op-ed in *TIME* magazine titled "Pausing AI Developments Isn't Enough. We Need to Shut it All Down." The op-ed: - Argued that the FLI pause letter was insufficient - Called for an international moratorium on large AI training runs, backed by the threat of airstrikes on AI data centers located outside the moratorium - Became one of the most controversial public AI safety statements of the 2023 period The *TIME* op-ed's reception was polarized — it was widely criticized as extreme by many observers while being defended by some AI safety researchers as a necessary escalation of public alarm. ## Leadership MIRI's leadership structure: - **Eliezer Yudkowsky**: Founder; Research Fellow and leading public voice - **Nate Soares**: Executive Director (2015-present); principal operational leader - **Malo Bourgon**: Chief Operating Officer - **Research staff**: ~10-15 researchers as of 2024, a smaller operation than 2010s peak ## Funding and Institutional Scale MIRI is a small operation by institutional standards: - **Annual budget**: ~$5-10 million - **Principal funders**: [[Peter Thiel]] (early and substantial), Open Philanthropy (sporadic grants), [[Jaan Tallinn]] (substantial), individual donors - **Staff**: 15-25 full-time across history MIRI's scale has always been deliberately modest; it has positioned itself as a specialist research organization rather than an institutional scale operation. ## Connection to the KB Ecosystem - **[[Eliezer Yudkowsky]]**: Founder; principal intellectual figure across MIRI's 25-year history. - **[[LessWrong]]**: Sister community; emerged from MIRI's intellectual and operational orbit. - **[[AI Safety]]**: MIRI is the founding institutional organization of the field. - **[[Existential Risk]]**: MIRI's work on AI x-risk is foundational. - **[[TESCREAL]]**: MIRI and the rationalist community around it are structurally part of the TESCREAL cluster. - **[[Nick Bostrom]]**: Intellectual colleague; Bostrom's *Superintelligence* substantially built on MIRI-era concepts. - **[[Peter Thiel]]**: Long-term funder. - **[[Jaan Tallinn]]**: Long-term funder. - **Rationalist community**: MIRI is institutionally central to the rationalist ecosystem that produced much of the EA movement's original recruitment base. ## See Also - [[Eliezer Yudkowsky]] - [[Nick Bostrom]] - [[LessWrong]] - [[AI Safety]] - [[Existential Risk]] - [[TESCREAL]] - [[Future of Humanity Institute]] ## Merged from “MIRI” (dedup pass 2026-06-11) > [!note] This section was merged from a duplicate page and awaits editorial integration. Formerly the Singularity Institute for Artificial Intelligence (SIAI). An AI safety research nonprofit founded in 2000 by [[Eliezer Yudkowsky]], focused on ensuring that artificial general intelligence (AGI) is developed safely — specifically, on the mathematical problem of "alignment" (ensuring a superintelligent AI's objectives match human values). [[Peter Thiel]] provided $100K in matching funds in 2006 and joined the advisory board, placing him at the origin point of the AI safety/existential risk community well before it became mainstream. ## The Thiel Dual-Positioning Pattern Thiel's early support for MIRI is significant because it shows him simultaneously investing in AI *deployment* (through [[Palantir]] and [[Founders Fund]] portfolio companies like DeepMind-precursor firms) while funding research on AI *risk*. This is a recurring Thiel pattern: - **Fund the thing**: Palantir (AI for surveillance), Founders Fund AI investments - **Fund the critique of the thing**: MIRI (AI might destroy civilization) - **Position yourself as essential to both sides**: become the person both builders and critics need to talk to The tension is not incidental — it is the structural source of influence. By sitting at the intersection of AI acceleration and AI safety, Thiel's network controls framing on both sides of the most consequential technology policy debate of the century. ## The Rationalist/EA Pipeline MIRI (as the Singularity Institute) was the institutional nucleus of what became the rationalist community — the intellectual culture that produced LessWrong, the effective altruism movement, and eventually key figures in AI policy debates at OpenAI, [[Anthropic]], DeepMind, and government AI safety offices. The **Singularity Summit** (co-organized by MIRI at Stanford, 2006–2012) was a major early forum for AI futurism. These events connected Thiel to the rationalist/effective altruist community and established relationships that would later prove politically consequential: - EA-aligned researchers staffed AI safety teams at the major labs - Rationalist-adjacent thinkers (e.g., those in the Yarvin orbit) overlapped with [[the Thiel network]] - The community's intense focus on existential risk from AI provided intellectual ammunition for Thiel-network arguments about AI governance ## The 2023 Pivot In 2023, MIRI announced a strategic pivot: Yudkowsky publicly stated that MIRI's technical research program had failed to make sufficient progress on alignment, and that the organization would shift toward policy advocacy — specifically, advocacy for slowing or pausing frontier AI development. This placed MIRI in opposition to the accelerationist wing of the AI industry, creating a fissure within communities that had previously aligned. The pivot matters for the KB because it represents a fracture within the Thiel-adjacent intellectual community: MIRI's "pause AI" position conflicts directly with Thiel-network companies (Palantir, Anduril) whose business models depend on rapid AI deployment. ## Relevance to the KB MIRI's primary significance is as a node in understanding how the Thiel network spans both AI *building* and AI *governance* — the same dual positioning visible in [[David Sacks]]'s role as "AI and Crypto Czar" overseeing regulation of companies his network funded. The pattern: fund the technology, fund the safety critique, then control the regulatory conversation that mediates between them. ## See Also - [[Peter Thiel]] - [[David Sacks]] - [[Thiel Foundation]] - [[Future of Life Institute]] - [[Open Philanthropy]] - [[Palantir]]