# Attention Mechanism The **attention mechanism** is the computational technique that allows AI models to selectively focus on relevant parts of their input when producing each part of their output. Introduced for machine translation (Bahdanau et al., 2014) and generalized into **self-attention** in the [[Transformer Architecture]] (Vaswani et al., 2017), attention is the core innovation that made modern AI possible. Before attention, neural networks processed sequences by compressing the entire input into a fixed-size representation — inevitably losing information, especially for long inputs. Attention solved this by allowing the model to look back at the full input at every step, weighting each element by its relevance to the current computation. ## How It Works For each position in the input, the model computes a set of **attention scores** — numerical weights indicating how much every other position matters for understanding the current one. In the sentence "The cat sat on the mat because it was tired," when processing "it," the attention mechanism assigns high scores to "cat" and lower scores to "mat" — learning the coreference without being explicitly programmed for it. In **multi-head attention** (the standard transformer approach), this process runs in parallel across multiple "heads," each learning to track different types of relationships — syntactic structure, semantic similarity, positional patterns, logical dependencies. A frontier model might run 96+ attention heads per layer across 100+ layers, creating an enormous capacity for capturing complex patterns. The cost: attention is **quadratic** in input length. Every token must attend to every other token, so doubling the [[Context Window]] quadruples the compute. This is why extending context windows from 4K to 1M+ tokens has required specialized techniques (sparse attention, sliding-window attention, ring attention, FlashAttention) and driven enormous hardware investment. ## KB Significance 1. **The mechanism of understanding.** When people ask "does AI really understand?" — the answer, to the extent it's yes, is located in attention. Attention heads learn to track grammar, coreference, reasoning chains, and factual associations. The patterns they discover are emergent — not programmed — and often surprising to researchers. 2. **The compute bottleneck.** Attention's quadratic cost is the technical reason behind the compute arms race. Longer context, deeper understanding, and more capable models all require more attention computation — which requires more GPUs, more energy, and more capital. The mechanism's elegance is inseparable from its expense. ## See Also - [[Transformer Architecture]] — The architecture built around attention - [[Context Window]] — The input-length constraint shaped by attention costs - [[Parameters]] — The learned weights that implement attention