# AI Ethics and Safety Ecosystem The network of organizations, researchers, funders, government bodies, and ideological factions that contest what AI safety means, who gets to define it, and whether safety is an existential priority or a competitive handicap. This ecosystem is where the technical question of "how do we make AI systems behave as intended" collides with the political question of "who decides what 'as intended' means." **Core tension:** The AI safety ecosystem is split between two camps that agree AI is dangerous but disagree about what the danger is and who should respond. The existential-risk camp (rooted in [[TESCREAL]], funded by [[Effective Altruism]]) focuses on catastrophic future scenarios — superintelligence, [[Recursive Self-Improvement]], loss of human control — and pushes for frontier labs to self-regulate through responsible scaling policies. The AI justice camp focuses on present harms — bias, surveillance, labor displacement, environmental cost, concentration of power — and pushes for democratic governance, regulation, and accountability. The two camps often talk past each other, and the frontier labs have exploited the split to position themselves as the responsible adults in the room while resisting external oversight from either direction. → For the frontier AI companies and their Pentagon relationships, see [[Silicon Valley AI Ecosystem]] → For defense applications specifically, see [[Defense Tech Ecosystem]] → For the ideological underpinnings, see [[TESCREAL]] --- ## The factions ### Existential risk / alignment camp The intellectual descendants of [[Eliezer Yudkowsky]]'s arguments about superintelligence risk. This faction treats AI alignment — ensuring advanced AI systems pursue human-intended goals rather than developing misaligned objectives — as the central problem of the century. Key claim: if we build [[AGI]] without solving alignment, the result could be human extinction or permanent disempowerment. **Key people:** - [[Eliezer Yudkowsky]] — MIRI founder, rationalist movement patriarch, AI doomer who has called for bombing datacenters if necessary - [[Geoffrey Hinton]] — "Godfather of deep learning," left Google in 2023 to warn about AI extinction risk - [[Ilya Sutskever]] — former OpenAI chief scientist, co-led the board coup against [[Sam Altman]], founded [[Safe Superintelligence]] - [[Dario Amodei]] — [[Anthropic]] CEO, left OpenAI over safety disagreements, architect of [[Constitutional AI]] and Responsible Scaling Policy - [[Max Tegmark]] — MIT physicist, [[Future of Life Institute]] co-founder, organized the March 2023 open letter calling for a six-month pause on giant AI experiments - [[Nick Bostrom]] — Oxford philosopher, author of "Superintelligence" (2014), founder of the [[Future of Humanity Institute]] - [[Jan Leike]] — former OpenAI alignment lead, resigned May 2024 citing safety being deprioritized, joined Anthropic **Key organizations:** - [[Machine Intelligence Research Institute]] (MIRI) — Yudkowsky's research institute, the original AI alignment research organization (founded 2000) - [[Future of Life Institute]] (FLI) — Tegmark's organization, organized the AI pause letter and the Asilomar AI principles - [[Future of Humanity Institute]] (FHI) — Bostrom's Oxford institute, closed April 2024 after 19 years - [[Centre for the Study of Existential Risk]] (CSER) — Cambridge-based, co-founded by Martin Rees and Jaan Tallinn - [[Centre for Effective Altruism]] (CEA) — organizational hub of the EA movement that funds much of the xrisk ecosystem - [[Open Philanthropy]] — the primary funder of AI safety research, EA-aligned, major backer of [[Anthropic]] - [[Anthropic]] — the frontier lab most identified with the safety agenda; Responsible Scaling Policy, [[Constitutional AI]], the company that got banned for refusing military use restrictions ### AI justice / accountability camp Researchers and advocates who argue that the existential-risk framing is a distraction from AI's present harms — and that the safety ecosystem's focus on hypothetical superintelligence conveniently deflects attention from the discrimination, surveillance, labor exploitation, and power concentration that current AI systems already produce. **Key people:** - [[Timnit Gebru]] — fired from Google AI Ethics in December 2020 over a paper on language model risks, founded the [[DAIR Institute]], coined "stochastic parrots" (with Emily Bender) - Emily Bender — University of Washington linguist, co-author of the "Stochastic Parrots" paper, critic of large language model hype - Joy Buolamwini — MIT researcher, founder of the Algorithmic Justice League, demonstrated racial bias in facial recognition systems - Meredith Whittaker — president of Signal, co-founder of AI Now Institute, former Google organizer - Safiya Noble — UCLA professor, author of "Algorithms of Oppression" **Key organizations:** - [[DAIR Institute]] — Gebru's Distributed AI Research Institute, founded after her Google firing - AI Now Institute — NYU-based research center focused on social implications of AI (founded by Whittaker and Kate Crawford) - Algorithmic Justice League — Buolamwini's organization focused on bias in AI systems - [[Black in AI]] — community and advocacy organization for Black researchers in AI - Data & Society — research institute studying social and cultural implications of data-driven technologies ### The frontier labs as self-regulators The major AI companies have positioned themselves between the two camps, adopting safety language while resisting external governance from either direction. - [[Anthropic]] — the lab most credibly committed to safety ([[Constitutional AI]], Responsible Scaling Policy, refusal to lift military-use restrictions that led to the [[Anthropic Government Ban]]). But the [[Anthropic Model Export Directive]] demonstrated that voluntary safety can become a liability: the company was punished for publicly releasing [[Claude Fable 5]] (where jailbreaks get discovered) while [[Claude Mythos 5]] (restricted to partners) was restored first - [[OpenAI]] — shifted from nonprofit safety mission to commercial dominance under [[Sam Altman]]; the November 2023 board crisis (where safety-aligned directors fired and then reinstated Altman) was the decisive moment where growth defeated governance. Jan Leike's resignation in May 2024 made the shift explicit - [[Google DeepMind]] — maintains safety research teams but accepted Pentagon "all lawful use" requirements without friction - [[xAI]] — [[Elon Musk]]'s lab, explicitly positioned against safety restrictions; Grok integrated into [[Project Maven]] for targeting in the [[Iran War 2026]] ### Government evaluation bodies - [[CAISI]] — NIST's Center for AI Standards and Innovation (formerly US AI Safety Institute); evaluates frontier models for dangerous capabilities, renamed from "safety" to "standards and innovation" under Trump - [[AI Security Institute]] — UK government's AI evaluation body (formerly AI Safety Institute); conducts pre-deployment testing, the US's primary international counterpart - [[EU AI Act]] — the most comprehensive AI regulatory framework, classifying AI systems by risk tier; the regulatory approach the US has explicitly rejected --- ## The funding architecture The AI safety ecosystem's funding reveals its political economy. The existential-risk camp is overwhelmingly funded by a small number of EA-aligned donors and foundations: - [[Open Philanthropy]] (Dustin Moskovitz / Cari Tuna) — the single largest funder of AI safety research; major early backer of [[Anthropic]] - Survival and Flourishing Fund — EA-aligned fund supporting xrisk research - Jaan Tallinn (Skype co-founder) — personal funder of CSER, FHI, MIRI, and other xrisk organizations The AI justice camp draws from more traditional philanthropic sources: the Ford Foundation, MacArthur Foundation, [[Open Society Foundations]], and academic institutions. The funding asymmetry matters: the xrisk camp has access to tech billionaire capital, while the justice camp operates on traditional nonprofit budgets. --- ## Key concepts in the vault The AI safety debate runs through a vocabulary of contested terms, most of which have dedicated pages: - [[AI Safety]] — the umbrella term, claimed by both camps - [[Alignment]] — ensuring AI systems pursue intended goals - [[Constitutional AI]] — Anthropic's approach to training models with constitutional principles - [[AI Doomerism]] — the stance that advanced AI will likely cause human extinction - [[Existential Risk]] — the broader framework of civilization-ending threats - [[Dual-Use AI]] — technology serving both civilian and military purposes - [[Frontier Model]] — the class of models at the capability boundary - [[Guardrails]] — technical constraints on model behavior - [[Alignment Tax]] — the performance cost of safety measures - [[Capability Overhang]] — latent potential exceeding current deployment - [[Recursive Self-Improvement]] — the compounding self-modification scenario - [[Scaling Laws]] — the empirical relationship between compute and capability - [[Emergent Capabilities]] — abilities appearing at scale without explicit training - [[AI Arms Race]] — the competitive dynamic driving safety shortcuts - [[Jailbreaking]] — bypassing model safety constraints --- ## The political map The AI safety ecosystem does not map onto conventional left-right politics in straightforward ways: The **xrisk/alignment camp** draws from rationalist and [[Effective Altruism]] communities that are politically heterogeneous but skew libertarian-technocratic. [[TESCREAL]] ideology (Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, Longtermism) provides the philosophical substrate. This camp has significant overlap with Silicon Valley wealth and, through EA's utilitarian framework, can rationalize both safety advocacy and acceleration depending on expected-value calculations. The **AI justice camp** is more straightforwardly progressive, rooted in civil rights, labor, and technology accountability traditions. Its critique of AI often extends to a critique of the power structures that build and deploy it — making it more threatening to the tech industry than the xrisk camp, which shares the industry's fundamental assumption that frontier AI development should continue (just more carefully). The **Trump administration** has rejected both camps' framing, treating AI governance as a competitiveness issue rather than a safety or justice issue. The renaming of the US AI Safety Institute to [[CAISI]] and the [[Anthropic Government Ban]] for maintaining safety restrictions on military use are the clearest signals: in this framework, safety is an obstacle to American AI dominance, and companies that prioritize it will lose government contracts to those that don't. --- ## Vault intersections - [[Silicon Valley AI Ecosystem]] — the companies this ecosystem tries to govern - [[Defense Tech Ecosystem]] — where AI capabilities meet military application - [[TESCREAL]] — the ideological substrate of the xrisk camp - [[Anthropic Government Ban]] — the case study in safety as competitive disadvantage - [[Anthropic Model Export Directive]] — the first AI-specific export control - [[Project Maven]] — the Pentagon AI program that crystallized the ethics debate - [[AI Lexicon]] — the full AI vocabulary - [[Global Intelligence Hub]] — intelligence agencies' role in AI evaluation ## See Also - [[Silicon Valley AI Ecosystem]] — the industry this ecosystem oversees - [[Defense Tech Ecosystem]] — the military applications driving the urgency - [[TESCREAL]] — the philosophical framework - [[Anthropic]] — the company most embedded in both safety and governance - [[Dual-Use AI]] — the framework bridging safety and security - [[EU AI Act]] — the regulatory approach - [[Ecosystems Hub]] — the full ecosystems index