# Mixture of Experts (MoE) An architecture where the model contains multiple specialized "expert" sub-networks, and a routing mechanism selects which experts to activate for each input. A MoE model with 1.8 trillion total parameters might activate only 200 billion for any given query — the router directs each input to the most relevant experts, and the inactive experts consume no compute. ## Why it matters MoE changed the meaning of "model size." GPT-4 reportedly uses a MoE architecture with ~1.8 trillion total parameters but only ~200 billion active per query. This means GPT-4 has the *knowledge* of a 1.8T-parameter model but the *compute cost* of a ~200B-parameter model — far cheaper to run at inference time than a dense model of equivalent total size. The architecture partially decouples [[Parameters]] from [[Inference vs Training]], breaking the simple narrative that bigger models = more expensive to run. It also complicates capability comparisons: a MoE model's total parameter count overstates its per-query complexity, while its active parameter count understates its total knowledge. For the [[Scaling Laws]] trajectory, MoE suggests that models can continue growing in total knowledge while keeping inference costs manageable — an important factor for the economic viability of frontier AI deployment. ## See Also - [[Parameters]] — the metric MoE complicates - [[Transformer Architecture]] — the base architecture MoE extends - [[Inference vs Training]] — the cost dimension MoE optimizes - [[Scaling Laws]] — the trajectory MoE enables