Hypothesis testing
The formal procedure for deciding whether data are surprising enough to reject a default assumption.
What it means
Hypothesis testing is the framework by which researchers use sample data to decide between competing claims about a population, typically by pitting a null hypothesis of 'no effect' or 'no difference' against an alternative. The standard procedure computes how surprising the observed data would be if the null were true—summarized by a p-value—and rejects the null when that probability falls below a pre-set threshold such as 0.05, while controlling the rates of two errors: false positives (Type I, wrongly rejecting a true null) and false negatives (Type II, missing a real effect). It is widely misunderstood: a p-value is not the probability that the hypothesis is true, statistical significance is not practical importance, and failing to reject the null is not proof that it holds. The approach is also vulnerable to abuse through p-hacking, selective reporting, and underpowered designs, abuses central to the replication crisis. Reforms include pre-registration, emphasis on effect sizes and confidence intervals, equivalence testing, and Bayesian alternatives. It matters because it remains the dominant logic by which behavioral claims are deemed evidentially supported.
Examples
To test whether a new teaching method works, a researcher assumes it makes no difference, runs a trial, and rejects that assumption only if the observed improvement would be very unlikely under pure chance.
A team assumes their redesigned checkout button makes no difference, runs it against the old one for a fortnight, and ships it only if a gap that large would be unlikely by chance.
A trial that misses significance gets reported as proof the drug does not work — though with only forty patients, a real and useful effect could easily have slipped past unnoticed.
First described in Fisher; Neyman & Pearson (1920s–1930s).