# Replication Crisis **The model:** The discovery, primarily since 2011, that a large proportion of published scientific findings — particularly in psychology, medicine, and social science — fail to replicate when other researchers attempt to reproduce them. This isn't a failure of individual studies; it's a systemic problem with how research is incentivized, conducted, analyzed, and published. ## Key Moments **Bem's ESP study (2011).** Daryl Bem published a study in a prestigious psychology journal claiming to find evidence for precognition — that people could sense future events. The study used standard psychological methods and passed peer review. The reaction catalyzed the crisis: if standard methods could produce evidence for ESP, maybe the methods themselves were broken. **Open Science Collaboration (2015).** A team of 270 researchers attempted to replicate 100 psychology studies published in top journals. Only 36% replicated successfully with a statistically significant result in the same direction. The average replicated effect size was half the original. This was the quantitative proof that the published literature was substantially unreliable. **Many Labs projects.** Large-scale replication efforts found that some classic effects replicated robustly across labs and cultures, while others — some taught in every introductory psychology course — evaporated entirely. The field had been teaching findings as facts that weren't. ## Causes **P-hacking.** Researchers make analytical choices (which variables to include, which outliers to remove, when to stop collecting data, which comparisons to report) until they find a P-value below 0.05. Each choice is defensible individually; collectively, they inflate false positive rates well beyond the nominal 5%. Also called "researcher degrees of freedom" or the "garden of forking paths." **Small sample sizes and low Statistical Power.** Underpowered studies can only detect large effects — and when an underpowered study does find a significant result, it's likely to be an overestimate (winner's curse). The published literature is disproportionately populated with underpowered studies that got lucky. **Publication Bias.** Positive results get published; null results don't. The published literature systematically overestimates effect sizes because the studies finding smaller or null effects are filtered out. **HARKing.** Hypothesizing After Results are Known — presenting post-hoc findings as if they were predicted a priori. A researcher runs 20 comparisons, finds one significant result, and writes the paper as if that comparison was the original hypothesis. This converts exploratory analysis into confirmatory-looking results. ## Reforms **Pre-registration.** Researchers publicly commit to their hypotheses, methods, and analysis plans before collecting data. This prevents p-hacking and HARKing by locking in the analytical plan. Registered Reports go further: journals accept or reject studies based on the methodology before results exist, eliminating publication bias. **Open data and materials.** Making datasets and code publicly available allows others to verify analyses and attempt replications. Errors that would have gone undetected in a closed system get caught. **Larger sample sizes.** Recognition that many fields were systematically underpowered has led to larger studies and multi-lab collaborations. **Revised statistical standards.** Some fields have proposed lowering the significance threshold from p < 0.05 to p < 0.005, or abandoning null-hypothesis significance testing altogether in favor of effect sizes, confidence intervals, or Bayesian methods. ## KB Applications ### Why "The Science" Is More Uncertain Than It Seems Political arguments that invoke "the science" — on policy topics from economics to public health to criminal justice — often cite published research as settled. The replication crisis reveals that a substantial fraction of that published research may not be reliable. This doesn't mean science is useless; it means that individual studies deserve skepticism, especially when they produce surprising or politically convenient results from small samples. ### Nutritional Science and Policy Whiplash Much of nutritional epidemiology has been particularly vulnerable to replication failure. Decades of dietary advice (low-fat diets, vitamin supplements, specific food avoidances) were based on observational studies with small effects, extensive p-hacking, and severe publication bias. The periodic reversal of nutritional advice ("eggs are bad / eggs are fine / eggs are bad again") is partly a replication crisis playing out in public. ### Social Priming and Policy Interventions "Nudge"-style behavioral interventions were heavily promoted for policy applications based on social psychology research. Many of the foundational priming studies have failed to replicate. Policies built on unreplicated findings — framing effects, anchoring manipulations, embodied cognition claims — may rest on weaker foundations than their advocates believe. ## See Also - P-value — the threshold that p-hacking exploits - Statistical Power — the design feature most studies lacked - Publication Bias — the filter that distorts the literature - Statistical Significance — the binary framework that enables the crisis