# Attention Is All You Need *Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, "Attention Is All You Need," arXiv:1706.03762 (12 June 2017); presented at NeurIPS 2017, pp. 5998–6008. Work done at [[Google]] Brain and Google Research, with Gomez at the University of Toronto.* ## What the Paper Is The paper that built the modern AI era. It proposes the **Transformer**, a neural-network architecture for turning one sequence into another (the original task was machine translation) that relies *solely* on an [[Attention Mechanism]] and throws out the two techniques that had dominated sequence modeling for a decade — recurrence (RNNs and LSTMs) and convolution. The deliberately cocky title is a claim about architecture: you do not need the recurrent machinery everyone assumed was essential; attention alone suffices. The practical payoff was speed. Recurrent models process a sentence one token at a time, so training cannot be fully parallelized. Self-attention compares every position in a sequence to every other position simultaneously, which maps cleanly onto GPUs. That single property — parallelizable training — is what later made it economically feasible to train models on a meaningful fraction of the internet. The [[Transformer Architecture]] is, in a real sense, the thing that made scaling laws bankable. ## Core Contributions **Self-attention.** Each token's new representation is a weighted blend of all tokens in the sequence, where the weights are learned from the content itself. This lets the model capture long-range dependencies (the subject of a sentence and its verb thirty words apart) in one step rather than passing information down a long chain. **Scaled dot-product attention.** The specific, cheap formula the paper uses to compute those weights — dot products of query and key vectors, scaled and softmaxed — chosen because it is fast and matrix-multiply-friendly. **Multi-head attention.** Rather than computing attention once, the model runs several attention operations in parallel ("heads"), each free to focus on a different kind of relationship (syntax, coreference, position). Their outputs are concatenated. **Positional encoding.** Because attention is order-blind by construction (it sees a *set*, not a *sequence*), the authors inject sinusoidal signals that encode each token's position, restoring the notion of word order without reintroducing recurrence. **Encoder–decoder stack.** The original Transformer pairs an encoder (reads the input) with a decoder (writes the output). Later families split this: BERT kept the encoder, the GPT line kept the decoder. On the benchmark tasks the architecture set new state-of-the-art translation scores (28.4 BLEU on WMT 2014 English–German, 41.8 on English–French) while training in a fraction of the time and compute of the recurrent and convolutional systems it beat. ## Why It Matters for the RWKB This paper is the technical floor beneath nearly everything the KB tracks in AI. It belongs in the [[Artificial Intelligence Timeline]] as the hinge between the "deep learning works" era and the "generative AI explosion" era. **It is the common ancestor.** [[OpenAI]]'s GPT line, [[Google]]'s BERT and Gemini, [[Anthropic]]'s Claude, Meta's Llama — all are Transformers. The architecture's dominance is why a single design choice made in one corporate lab in 2017 now shapes the entire industry, the AI-safety debate, and the geopolitics of compute. **It is a study in corporate-captured fundamental research.** The breakthrough that defined a trillion-dollar industry was produced inside [[Google]] and published openly. Google then largely failed to commercialize its own invention first — the lesson every Big Tech lab absorbed, and a reason the field swung hard toward closed, proprietary models afterward. **The authors scattered — and seeded the private AI economy.** None of the eight stayed at Google long. The diaspora is itself an ecosystem map: - **Aidan Gomez** co-founded **Cohere** (enterprise LLMs). - **Ashish Vaswani** and **Niki Parmar** went to **Adept AI**, then co-founded **Essential AI**. - **Noam Shazeer** founded **Character.AI** (2021), then returned to Google in 2024 in a ~$2.7B licensing-and-talent deal that functioned as an acqui-hire. - **Illia Polosukhin** left to build **NEAR Protocol**, a blockchain/crypto platform — a direct bridge between the transformer story and the KB's [[Cryptocurrency Ecosystem]] coverage. - **Jakob Uszkoreit** co-founded **Inceptive**, applying the architecture to mRNA medicine design. - **Llion Jones** co-founded **Sakana AI** in Tokyo. - **Łukasz Kaiser** joined **OpenAI**. The pattern — fundamental research done on a corporate payroll, then privatized into a fleet of competing labs the moment it proved valuable — recurs throughout the KB's account of how AI consolidated. See [[Project Maven]] for the parallel story of how Google's AI capability flowed toward the Pentagon, and the [[Artificial Intelligence Timeline]] for the full arc. ## Limitations and Caveats - The paper's framing is narrow by design — it is a translation-benchmark paper. Almost none of what made Transformers world-historical (scaling laws, in-context learning, emergent capabilities, alignment problems) is anticipated in the text. Its significance is overwhelmingly *downstream* of what it actually claims. - "Attention" here is a precise mathematical operation, not the cognitive or economic sense of the word used elsewhere in the vault. Do not conflate it with the [[Attention Economy]]. - Crediting the modern AI era to one paper flattens real prior work: attention mechanisms existed before 2017 (Bahdanau et al., 2014), and the Transformer built on a decade of neural-MT research. The paper's contribution was removing recurrence and showing attention *alone* sufficed — consequential, but not virgin birth. ## Citation Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. *Advances in Neural Information Processing Systems, 30*, 5998–6008. arXiv:1706.03762. https://arxiv.org/abs/1706.03762 ## See Also - [[Attention Mechanism]] — the operation the paper is built on - [[Transformer Architecture]] — the architecture the paper introduced - [[Large Language Model]] — the technology the architecture made possible - [[Artificial Intelligence Timeline]] — where this paper anchors the Transformer Revolution era - [[Project Maven]] — the parallel Google-to-Pentagon AI story - [[OpenAI]] · [[Anthropic]] · [[Google DeepMind]] — the labs built on this architecture