# Scaling Laws **Scaling laws** are empirical relationships showing that AI model performance improves predictably as you increase three variables: the amount of training data, the number of model parameters, and the amount of compute used for training. First systematically documented by OpenAI researchers (Kaplan et al., January 2020) and refined by DeepMind's "Chinchilla" paper (Hoffmann et al., March 2022), scaling laws transformed AI development from an experimental science into something closer to engineering: if you know how much compute you have, you can predict roughly how capable the resulting model will be. The core finding is that performance follows a **power law** — smooth, predictable improvement on a log-log scale. Double the compute, and you get a measurable, predictable improvement. This relationship has held across orders of magnitude, from millions to trillions of parameters, and across modalities (text, images, code, video). ## The Two Eras of Scaling **Kaplan scaling (2020):** OpenAI's original paper found that model size (parameters) mattered most — you should make models as large as possible and train them on "enough" data. This philosophy drove the jump from GPT-2 (1.5B parameters) to GPT-3 (175B parameters): make it bigger. **Chinchilla scaling (2022):** DeepMind showed that Kaplan's approach was inefficient. The optimal strategy is to scale data and parameters together — a model with 70B parameters trained on 1.4 trillion tokens outperformed a 280B-parameter model trained on fewer tokens. This realization shifted the bottleneck from model size to data availability, triggering the scramble for training data that led to mass web scraping, publisher licensing deals, and synthetic data generation. **Post-Chinchilla (2024–present):** "Inference-time scaling" or "test-time compute" — the discovery that you can improve performance by giving models more compute at inference (thinking longer, generating and evaluating multiple solutions) rather than only at training. OpenAI's o1/o3 and Anthropic's extended-thinking models exploit this, partially decoupling capability from training cost and shifting it to [[Inference vs Training]]. ## The Political Economy of Scaling Scaling laws are not just a technical finding — they are the **economic engine** of AI's capital concentration. The logic is simple and self-reinforcing: **More compute → better models → more revenue → more capital → more compute.** This creates a flywheel that only organizations with access to enormous capital can ride. Training cost for [[Frontier Model]] has increased roughly 4× per year: GPT-3 cost ~$10M (2020), GPT-4 cost ~$100M (2023), and current frontier training runs exceed $1B (2026). Each generation requires more GPUs, more energy, more data, and more money — and each generation produces models sufficiently better to justify the expenditure. The result is that scaling laws function as a **barrier to entry** that grows higher with every generation. The number of organizations capable of training at the frontier is shrinking. Scaling laws are the mechanism by which AI development concentrates into an oligopoly. ## The "Scaling Hits a Wall" Debate Throughout 2024–2025, a recurring debate emerged about whether scaling laws were "hitting a wall" — whether the returns from simply adding more compute were diminishing. Evidence cited included: GPT-4.5's incremental improvement over GPT-4, the difficulty of finding new high-quality training data, and the growing cost of each marginal improvement. The response from frontier labs has been twofold: first, inference-time scaling partially sidesteps training-compute limitations; second, architectural innovations (mixture-of-experts, retrieval augmentation, agentic workflows) create new scaling dimensions beyond raw parameter count. DeepSeek's R1 (January 2025) complicated the narrative further by demonstrating frontier-class reasoning from a much smaller model trained with novel efficiency techniques on older hardware — suggesting that the scaling curve can be bent by algorithmic innovation, not just brute compute. The political significance: if scaling laws hold indefinitely, power concentrates further with each generation. If they plateau, the frontier becomes accessible to more players — including open-weight projects, smaller nations, and Chinese labs operating under U.S. GPU export controls. ## KB Significance 1. **The economic logic of concentration.** Scaling laws explain *why* AI development is concentrating into fewer hands. It's not conspiracy — it's the mathematics of the technology. Each generation costs more, and only organizations with billions in capital can play. The KB's documentation of tech-oligarchy power structures rests on this economic foundation. 2. **The arms-race accelerant.** The predictability of scaling laws turns AI development into a race where the prize goes to whoever can deploy the most compute. This predictability is what makes the U.S.-China AI competition feel existential to policymakers: both sides believe that falling behind on the scaling curve means falling behind permanently. The GPU export controls, the [[The Quest for 90 Gigawatts]], and the trillion-dollar capital commitments are all downstream of this belief. 3. **The DeepSeek challenge.** DeepSeek's R1 demonstrated that algorithmic efficiency can partially substitute for raw compute — raising the question of whether U.S. export controls on GPUs are an effective strategy if Chinese labs can achieve comparable results with less hardware. Scaling laws describe the *default* trajectory; whether they can be circumvented through innovation is the central question of AI geopolitics. ## See Also - [[Foundation Model]] — What scaling laws are applied to - [[Frontier Model]] — The most expensive products of scaling - [[Transformer Architecture]] — The architecture that made scaling possible - [[Compute Sovereignty]] — Who has the hardware to scale - [[Inference vs Training]] — The new dimension of scaling - [[Neurosymbolic AI]] — The rival thesis: structure, not scale, is the missing ingredient