Statistical significance

What it means

Statistical significance is a verdict that a result is unlikely to have arisen by chance alone, declared when the p-value — the probability of observing data at least as extreme as those obtained if the null hypothesis were true — falls below a pre-set threshold, conventionally 0.05. The logic is one of proof by improbability: if the data would be very surprising in a world with no real effect, that world is rejected in favor of the alternative. The concept is, however, one of the most widely misunderstood ideas in science, and several common interpretations are simply wrong: the p-value is not the probability that the hypothesis is true, not the probability the result is a fluke, and not a measure of the size or importance of an effect. The 0.05 threshold is an arbitrary convention, and significance can be achieved for trivially small effects given a large enough sample, or missed for real effects in an underpowered one. Critics have proposed lowering thresholds, reporting confidence intervals and effect sizes instead, or abandoning significance testing altogether. It matters because significance is treated as a gatekeeper for publication and decisions, yet on its own it says nothing about whether an effect is large, real in a practical sense, or replicable.