# Transformer Architecture The **transformer** is the neural network architecture underlying virtually every major AI system in 2026 — GPT, Claude, Gemini, Llama, [[Stable Diffusion]], Sora, and hundreds of others. Introduced in the 2017 paper "Attention Is All You Need" by researchers at Google Brain and Google Research, the transformer replaced earlier sequential architectures (RNNs, LSTMs) with a mechanism called **self-attention** that processes all parts of an input simultaneously rather than one token at a time. This parallelism was the key unlock. Previous architectures processed text like reading a book one word at a time, making training slow and limiting how much context the model could retain. Transformers process the entire input at once, allowing the model to learn relationships between any two elements regardless of distance — and, critically, allowing training to scale across thousands of GPUs in parallel. Without the transformer, the scaling revolution documented in [[Scaling Laws]] would not have been possible. ## How Self-Attention Works (Simplified) For each element in the input (a word, an image patch, a code token), the transformer computes three things: a **query** (what am I looking for?), a **key** (what do I contain?), and a **value** (what information do I carry?). Every element's query is compared against every other element's key to produce attention scores — a map of which parts of the input are relevant to which other parts. These scores weight the values, determining how much each element "pays attention to" every other element. In practice, this happens across multiple **attention heads** in parallel, each learning to track different types of relationships (syntactic, semantic, logical, positional). A large model might have 96+ attention heads per layer across 100+ layers — creating an enormous capacity for learning patterns. The compute cost of self-attention scales quadratically with input length (every token attends to every other token), which is why [[Context Window]] expansion from 4K to 1M+ tokens has required architectural innovations (sparse attention, ring attention, sliding-window attention) and enormous hardware investment. ## The Irony of Origin The transformer was invented at Google — by eight researchers, several of whom have since left for competitors or founded their own companies. The "Attention Is All You Need" paper is one of the most consequential technical publications in history, and Google initially underexploited it. OpenAI's GPT series (Generative Pre-trained Transformer) took the architecture and scaled it aggressively; Anthropic was founded by former OpenAI researchers who helped build that scaling pipeline. Google's invention of the transformer and subsequent loss of the early lead illustrates a recurring pattern in AI: the research lab that makes the breakthrough is not necessarily the organization that captures the value. The transformer's commercial value has been captured primarily by OpenAI (via Microsoft's investment), Anthropic (via Amazon/Google investment), and Meta (via Llama's ecosystem play) — not by the Google research team that invented it. ## KB Significance 1. **The enabling technology.** Every AI system documented in this KB — from [[Palantir AIP]] to Pentagon propaganda generators to the [[Large Language Model]] powering content automation — runs on transformer architecture. The transformer is to AI what the transistor was to computing: the fundamental unit that made everything else possible. 2. **The compute multiplier.** The transformer's parallelism is what made the scaling era possible, which in turn created the GPU shortage, the NVIDIA monopoly, the data-center buildout, and the [[The Quest for 90 Gigawatts]]. The architecture's appetite for parallel computation is the technical root of the capital concentration documented across the KB. 3. **The talent bottleneck.** The small number of people who deeply understand transformer architecture at the frontier level — and who know how to train them at scale — is a key chokepoint in the AI ecosystem. This talent concentration reinforces the organizational concentration: only labs that can attract and retain transformer-scaling expertise can compete at the frontier. ## See Also - [[Large Language Model]] — The dominant application of transformer architecture - [[Foundation Model]] — The paradigm transformers enabled - [[Scaling Laws]] — Why transformers keep getting bigger - [[Attention Mechanism]] — The core innovation inside transformers - [[Context Window]] — The input-length constraint shaped by transformer compute costs