Researcher degrees of freedom
The many small analytic choices that quietly let you find what you want.
What it means
Researcher degrees of freedom are the numerous, often defensible decisions an analyst makes when collecting and analyzing data — which observations to exclude, which covariates to include, how to code variables, when to stop collecting, which outcomes to report. Each choice seems innocuous, but the freedom to make them after seeing the data lets a researcher, even unwittingly, steer results toward significance. Exploited across a whole analysis, this flexibility can push the false-positive rate far above the nominal 5%. Recognizing it motivated transparency reforms: pre-registration, full reporting of conditions and exclusions, and robustness checks across reasonable specifications.
Examples
Choosing, only after peeking, to drop three 'outliers' and add a covariate because doing so nudges p below 0.05.
Checking an A/B test every morning and stopping the moment it crosses significance turns a coin-flip difference into a 'winner' that the product team then ships.
A trial measures ten outcomes, finds the one that happens to move, and the paper is written as though that outcome had been the point all along.
First described in Simmons, Nelson & Simonsohn (2011).