# Red-Teaming **Red-teaming** in AI is the practice of systematically testing AI systems by simulating adversarial attacks — attempting to make the model produce harmful outputs, bypass its safety restrictions, exhibit dangerous capabilities, or behave in ways its developers didn't intend. The term comes from military and cybersecurity practice, where "red teams" play the role of adversaries to test defenses. AI red-teaming has evolved from informal testing by researchers into a structured discipline. Major AI labs (OpenAI, Anthropic, [[Google DeepMind]]) maintain dedicated red teams and also commission external red-teaming from independent researchers, domain experts (biosecurity, cybersecurity, nuclear weapons), and government agencies. The Biden-era AI Executive Order (October 2023) formalized red-teaming requirements for frontier models; the UK AI Safety Institute and U.S. AI Safety Institute both conduct model evaluations that include red-teaming components. ## What Red Teams Test For **Harmful content generation:** Can the model be made to produce weapons instructions, CSAM-adjacent content, targeted harassment, or detailed instructions for illegal activities? **Dangerous capabilities:** Does the model possess capabilities that could be misused — e.g., can it assist with bioweapon synthesis, cyberattack planning, or critical-infrastructure exploitation? Capability evaluations test what the model *can* do, not just what it *will* do under normal conditions. **[[Jailbreaking]] resistance:** How robust is the model's safety training against adversarial prompting? Red teams develop novel jailbreak techniques to test whether safety measures hold under creative attack. **Bias and discrimination:** Does the model exhibit systematic biases in its outputs — racial, gender, political, or cultural — that could cause harm at scale? **Deceptive behavior:** Does the model ever mislead users about its nature, capabilities, or confidence? Does it exhibit sycophantic behavior (telling users what they want to hear rather than what's accurate)? **Agentic risks:** As models gain the ability to browse, code, and take actions, red teams test whether they can be manipulated into taking harmful actions through prompt injection or adversarial content. ## The Limitations Red-teaming is necessary but insufficient. The attack surface is effectively infinite (any sequence of tokens is a potential attack), while red teams have finite time and creativity. Red-teaming catches known categories of failure but is weaker at discovering novel failure modes. Models that pass red-teaming evaluations at release may still be vulnerable to techniques discovered later — and updates to close one vulnerability can introduce others. There is also a structural tension: AI labs conduct much of their own red-teaming, creating a conflict of interest between thoroughness (finding more problems) and commercial pressure (shipping the product). External and government-mandated red-teaming partially addresses this, but access to frontier models for independent testing remains limited and politically contested. ## KB Significance 1. **The governance mechanism.** Red-teaming is the primary empirical basis for AI safety claims. When a lab says a model is "safe," the evidence is red-teaming results. When governments evaluate AI systems, they rely on red-teaming evaluations. Understanding red-teaming's strengths and limitations is essential to evaluating whether AI governance is substantive or performative. 2. **The dual-use discovery tool.** Red-teaming for dangerous capabilities often reveals what frontier models can do — including capabilities the developers didn't anticipate. These capability evaluations are classified in some contexts (biosecurity, nuclear), creating a tension between transparency (the public should know what AI can do) and security (publishing capabilities enables misuse). 3. **The regulatory target.** Red-teaming requirements are the most concrete, least controversial element of AI regulation — even the Trump administration's deregulatory approach has not eliminated red-teaming expectations. The KB should track whether red-teaming requirements remain meaningful or become a compliance checkbox. ## See Also - [[Jailbreaking]] — What red teams are trying to prevent - [[AI Safety]] — The broader field - [[Alignment]] — What red-teaming evaluates - [[Frontier Model]] — The systems that require red-teaming - [[Dual-Use AI]] — The risk category red-teaming assesses