p-hacking
Also known as: Data dredging
Tweaking analyses until a result crosses the significance threshold.
What it means
p-hacking, or data dredging, is the practice — conscious or not — of exploiting the many flexible choices available in data analysis until a result reaches statistical significance, then reporting only that result. The mechanism is researcher 'degrees of freedom': analysts can try multiple outcome measures, add or drop covariates, exclude 'outliers' under various rules, test subgroups, and decide when to stop collecting data, and each undisclosed choice is effectively an extra roll of the dice that inflates the chance of a false positive far above the nominal 5%. Because only the analysis that 'worked' is published, the resulting finding is likely to be a fluke that will not replicate, which makes p-hacking a central driver of the replication crisis. It is distinct from outright fabrication and often arises from motivated reasoning and the pressure to publish significant results, rather than from deliberate deception. The recognized defenses are pre-registration of the analysis plan, transparent reporting of all measures and exclusions, correction for multiple comparisons, and a shift in emphasis from significance to effect size and replication. It matters because it manufactures persuasive but spurious findings that waste resources and corrode trust in the literature.
Examples
Dropping 'outliers,' adding covariates, and testing one subgroup after another until the p-value finally falls below 0.05, then reporting only that test.
A growth team peeks at their A/B test every morning and stops the moment the new button color crosses significance — the repeated checking, not the button, produced the win.
A supplement trial measures fifteen health markers, finds one improved at p = 0.04, and the press release announces that finding alone; the other fourteen are never mentioned.
First described in Simmons, Nelson & Simonsohn (2011).