Confidence interval
A range of plausible values for a quantity, with a stated coverage rate.
What it means
A confidence interval is a range computed from data such that, over many repeated samples, a stated percentage of such intervals would contain the true parameter. Its width conveys the precision of an estimate, widening with noise and narrowing with larger samples, and it carries far more information than a bare 'significant or not' verdict. The standard interpretation is subtle and frequentist: the 95% refers to the procedure's long-run performance, not to a 95% probability that this specific interval traps the truth. Reporting intervals shifts attention from the existence of an effect to its likely magnitude and uncertainty, which is why many fields now prefer them to p-values alone.
How the width is set
The usual recipe is an estimate plus or minus a margin: the point estimate, a critical value from a reference distribution (roughly 1.96 for a 95% normal interval), and the standard error, which shrinks with the square root of the sample size. So quadrupling the sample only halves the width. Three things widen an interval: noisier data, smaller samples, and demanding higher confidence, since a 99% interval must succeed more often than a 95% one. The interval is symmetric only when the sampling distribution is; proportions near zero or one, ratios, and small counts often need asymmetric methods. None of this touches systematic error. A biased measurement produces a tight interval centered confidently on the wrong value.
The interpretations that go wrong
Surveys find the frequentist reading is widely missed. Hoekstra and colleagues (2014) gave researchers and students six statements about a 95% interval, all of them false, and every group endorsed several, with experience offering little protection; a replication by Garcia-Perez and Alcala-Quintana (2016) largely confirmed the pattern while disputing some wording. The seductive errors are consistent: treating the interval as having a 95% probability of holding the true value, reading its width as the probability the estimate is close, or thinking values inside are likely and those outside are ruled out. Each imports a Bayesian meaning the procedure does not supply. The 95% describes how often the method works across hypothetical repetitions, not the standing of the one interval in front of you.
What coverage does not guarantee
Morey and colleagues (2016) push further: even a procedure with exact 95% coverage need not yield intervals whose contents are the plausible parameter values. They construct cases where an interval is guaranteed to contain the truth, or guaranteed to miss it, yet still reports 95%, because coverage is a property of the recipe before the data arrive, not a statement about the numbers in hand. Coverage also assumes the model is right and the sample random; under a misspecified model or selective reporting the stated rate is fiction. And an interval quantifies sampling noise alone. Confounding, measurement error, and non-response sit outside it entirely, which is why a narrow interval is not the same as an accurate one.
Confidence versus credible intervals
The interval people usually want is the Bayesian credible interval, which does make a direct probability claim: given the data and a prior, there is a 95% probability the parameter lies inside. The two can coincide numerically, and with large samples and flat priors they often nearly do, but they answer different questions, and treating a confidence interval as if it were credible is the core confusion above. A prediction interval is different again: it brackets where a single future observation should fall, and is always wider than the interval for the mean, because it must absorb the spread of individuals rather than only the uncertainty in an average.
Examples
An estimated effect of 4 points with a 95% interval of [1, 7] signals both that an effect is likely and how imprecisely it is known.
A poll puts one party on 41% with an interval of 38 to 44 and its rival on 39% with 36 to 42 — the reported two-point lead is thinner than the noise around it.
An A/B test finds the new checkout lifts conversion by 0.2%, interval −1.1% to 1.5%. The honest read is not 'no effect' but 'this test was far too small to tell.'
A drug trial reports a hazard ratio of 0.85 with a 95% interval of 0.70 to 1.03. Because the range crosses 1, the data are compatible with a benefit and with no effect at all.
A factory samples 200 bolts and estimates their mean length with a 95% interval of 49.8 to 50.1 mm; the narrow band shows the average length is pinned down tightly, but it describes the mean, not the scatter of individual bolts.
First described in Jerzy Neyman (1937).
Key references
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. doi.org/10.1007/s10654-016-0149-3
- Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23(1), 103-123. doi.org/10.3758/s13423-015-0947-8
- Garcia-Perez, M. A., & Alcala-Quintana, R. (2016). The interpretation of scholars' interpretations of confidence intervals: Criticism, replication, and extension of Hoekstra et al. (2014). Frontiers in Psychology, 7, 1042. doi.org/10.3389/fpsyg.2016.01042
- Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21(5), 1157-1164. doi.org/10.3758/s13423-013-0572-3
- Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society A, 236(767), 333-380. doi.org/10.1098/rsta.1937.0005