Bootstrapping
Also known as: Bootstrap resampling
Estimating uncertainty by resampling your own data, over and over.
What it means
Bootstrapping is a resampling method that approximates the sampling distribution of a statistic by repeatedly drawing samples, with replacement, from the observed data and recomputing the statistic each time. The spread of these bootstrap estimates yields standard errors and confidence intervals without relying on formulas that assume a particular distribution. This makes it invaluable for complex statistics, like medians, ratios, or correlations, where analytic results are intractable. Its key assumption is that the sample adequately represents the population, so the technique inherits any bias in the original data and can perform poorly for very small samples or extreme quantities.
The plug-in principle
The method rests on a single substitution. The population you cannot see is replaced by the sample you can, treated as a miniature population in which each observed value carries equal probability. Drawing with replacement from it mimics drawing fresh samples from the world: because a value can appear twice or not at all, each resample differs, and the statistic wobbles from one resample to the next exactly as it would across real repeated studies. That wobble, mapped out over thousands of resamples, is the estimated sampling distribution. Everything downstream, a standard error, a bias estimate, a confidence interval, is read off it. The formula that classical statistics would derive by calculus is replaced with brute-force computation, which is why the technique arrived only once computing became cheap.
When the theory backs it
The substitution is not merely intuitive; for smooth statistics it is provably sound. Bickel and Freedman (1981) and Singh (1981) showed that as the sample grows, the bootstrap distribution of a statistic like the mean converges to its true sampling distribution, because the empirical distribution converges to the real one. For many well-behaved quantities the bootstrap is even second-order accurate, tracking the skew of the true distribution more faithfully than the textbook normal approximation and giving intervals whose coverage error shrinks faster with sample size. This accuracy is not automatic. It holds when the statistic depends smoothly on the data, and it is precisely that smoothness that some estimators lack, which is where the method starts to fail.
The interval you build matters
Not all bootstrap confidence intervals are equal, and the choice matters more than practitioners expect. The naive percentile interval simply reads the 2.5th and 97.5th quantiles off the resampled estimates. It is intuitive but only first-order accurate and blind to bias, so it can be systematically off when the statistic's distribution is skewed. DiCiccio and Efron (1996) developed corrected intervals, the bias-corrected-and-accelerated (BCa) and bootstrap-t, that adjust for bias and skewness and are second-order accurate. Counterintuitively, Hesterberg (2015) shows the popular pairing of a nonparametric bootstrap with a percentile interval is less accurate than an ordinary t-interval for small samples, and more accurate only once samples grow. The resampling is the easy part; picking the interval method is where accuracy is won or lost.
Where it breaks down
The bootstrap inherits the data's limits and fails outright when its smoothness assumption is violated. Estimating the maximum of a distribution, or any parameter living at the edge of the data, is inconsistent under the ordinary bootstrap: a resample can never exceed the largest value observed, so the method cannot see past it (Bickel and Freedman 1981). Very small samples give it too little to work with, since resampling eight numbers only reshuffles those eight and cannot manufacture information the study never collected. Dependent data breaks it differently: resampling rows independently destroys the correlation in a time series or the clustering in grouped data, so block and cluster variants are required. When the naive method stumbles, the m-out-of-n bootstrap, which draws fewer points than the sample size, often restores consistency.
Examples
Resampling 10,000 times from a dataset to build a confidence interval for the median, which has no simple formula.
In an A/B test where a few big spenders dominate revenue, no tidy formula covers the uncertainty in average spend, so the analyst resamples users thousands of times and reads the interval off the spread.
A conservation team with sixty tagged turtles resamples its own catch repeatedly to put error bars on a population ratio, rather than trusting a formula assuming a bell curve the data lack.
Bagging trains each decision tree in a random forest on its own bootstrap resample of the rows; averaging trees fit to slightly different data lowers variance and stabilizes the predictions.
To put error bars on a Gini coefficient of household income, an economist resamples the survey respondents ten thousand times, since the statistic's sampling distribution has no tidy closed form.
First described in Bradley Efron (1979).
Key references
- Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371-386. doi.org/10.1080/00031305.2015.1089789
- DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228. doi.org/10.1214/ss/1032280214
- Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. doi.org/10.1201/9780429246593
- Bickel, P. J., & Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. The Annals of Statistics, 9(6), 1196-1217. doi.org/10.1214/aos/1176345637
- Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26. doi.org/10.1214/aos/1176344552