When the Bootstrap Fails

The ordinary bootstrap works by treating your one sample as a stand-in for the entire population and repeatedly re-drawing from it with replacement, using the spread of the resulting statistics as an estimate of how uncertain the original statistic is. This is like trying to learn what's inside a 500-page book by re-reading the same 20-page pamphlet over and over. That trick works remarkably well when the pamphlet is a genuinely representative, independent, well-behaved excerpt of the book. It fails in three specific, predictable ways: when the pamphlet's pages are glued together in an order that matters (the data has dependence, like autocorrelated daily returns), when the book contains a chapter type your pamphlet never happened to sample (the data is heavy-tailed, with rare extreme events), or when the pamphlet is simply too short to contain the answer at all (the sample is very small). Re-shuffling the same limited material can't manufacture information the original sample never captured — and financial return data is, almost by default, exactly the kind of material where this bites.

Failure mode 1: dependence

The ordinary bootstrap draws each resampled observation independently with replacement from the original data. If the original data has dependence — daily returns show volatility clustering and short-run autocorrelation — resampling individual days scrambles that structure, because each resampled day is drawn without any memory of which day came before it in the original series. The result: any statistic whose true uncertainty depends on the dependence structure (like the standard error of a Sharpe ratio, which is inflated by positive return autocorrelation) gets a bootstrap-estimated standard error that's too small, because the resampling has silently destroyed the very dependence that was supposed to be part of the uncertainty.

The explorer above samples paths from a process with genuine memory — each step depends on where the path already is, unlike a plain i.i.d. sequence. Real daily returns have exactly this kind of short-run dependence (volatility clustering, momentum), which is precisely the structure an ordinary i.i.d. bootstrap destroys by resampling single days independently, as the next example shows numerically.

Worked example 1: autocorrelation collapses under naive resampling

Take a short 10-day return series with an obvious momentum pattern (in percent): $[0.5, 0.6, 0.4, -0.3, -0.5, -0.4, 0.7, 0.8, 0.6, -0.2]$ — up days cluster together, down days cluster together, giving this series a visibly positive lag-1 autocorrelation. Computing the sample lag-1 autocorrelation on the original series (correlation between the series and itself shifted by one day) comes out clearly positive, roughly in the $0.5$-$0.7$ range given how tightly the signs cluster. Now draw an ordinary i.i.d. bootstrap resample — 10 days drawn independently, with replacement, from the same 10 values. Because each draw is independent of the one before it, the expected lag-1 autocorrelation of the resampled series is zero, regardless of how strong the autocorrelation was in the original — resampling has thrown away exactly the structure being measured. A block bootstrap, resampling contiguous 3-day blocks instead of single days (e.g. drawing blocks like $[0.5,0.6,0.4]$ or $[-0.5,-0.4,0.7]$ as intact units and stitching several blocks together to rebuild a 10-day series), preserves the within-block momentum pattern, so the resampled series' autocorrelation stays much closer to the original's — which is why the reported standard error of something like a Sharpe ratio, computed from block-bootstrap resamples, is noticeably wider (more honest) than the artificially tight one an i.i.d. bootstrap would report.

Failure mode 2: tails and extreme quantiles

Worked example 2: the bootstrap can't invent a worse outcome than it's already seen

Take a small sample of 8 daily returns containing one clear extreme value: $[0.2, -0.1, 0.3, 0.1, -0.2, 0.4, -0.3, -4.5]$ — the last value, a $-4.5\%$ day, is by far the most extreme observation. Every possible i.i.d. bootstrap resample is built purely by drawing (with replacement) from this same set of 8 numbers — there is no mechanism by which a resample can ever produce a value smaller (more extreme) than $-4.5$, because $-4.5$ is literally the smallest number available to be drawn. So the bootstrap-estimated sampling distribution of the sample minimum has a hard floor at $-4.5$, and any confidence interval for an extreme quantile or a tail-risk statistic built from these resamples is mechanically incapable of reflecting the possibility of an outcome worse than what has already been observed — even though, in reality, a return series that has already produced one $-4.5\%$ day plausibly has some chance of eventually producing something worse. The bootstrap doesn't know that; it can only recombine what it's already seen.

What this means in practice

• Sharpe ratio and drawdown confidence intervals need a dependence-aware bootstrap. Because daily and even monthly returns show volatility clustering and short-run autocorrelation, an unadjusted i.i.d. bootstrap on a return series routinely understates the true uncertainty of performance statistics — block, stationary, or sieve bootstraps exist specifically to fix this.
• Backtest significance testing is exposed to the same issue. A backtested strategy's returns inherit whatever dependence structure the underlying market data has, so bootstrap-based significance tests on backtest Sharpe ratios need the same dependence-aware treatment, not the textbook i.i.d. version.
• Tail-risk and VaR-type statistics generally call for extreme value theory, not a naive bootstrap. Because the bootstrap can't extrapolate past the worst observation already in the sample, estimating how bad a 1-in-1,000-day event could be is fundamentally not a resampling problem — it needs a model (like a generalized Pareto tail fit) that extrapolates beyond the observed data.
• Very small samples are a red flag regardless of dependence. If the original sample is too short to be a representative excerpt of the underlying process, no amount of resampling recovers the missing representativeness — the bootstrap amplifies whatever information is present, it doesn't add new information.