Optional stopping
Also known as: Data peeking, Sequential testing problem
Repeatedly checking results and stopping when they turn significant inflates errors.
What it means
Optional stopping is the practice of analyzing data as they accumulate and ending collection as soon as a desired result, such as p < 0.05, appears. Because each interim look is another chance for noise to cross the threshold, this drastically inflates the false-positive rate above its nominal level, even when the analyst tries only a few peeks. It is one of the most potent and least visible forms of p-hacking, since the final dataset bears no trace of the looks that preceded it. Legitimate interim analysis exists, but it requires pre-planned sequential designs with adjusted thresholds that spend the error budget across looks.
Examples
Adding participants in batches and stopping the study the first time the p-value dips below 0.05.
A growth team watches the A/B dashboard hourly and ships the new button the first afternoon it goes green. The lift evaporates next quarter: they stopped on a wave, not a tide.
A supplement firm runs its trial with a live results feed and calls a halt the week the benefit turns significant. Nothing in the published dataset records the eleven earlier weeks when it didn't.
First described in Sequential analysis (Wald, 1947); flagged by Simmons, Nelson & Simonsohn (2011).