Pairs Versus Residual Bootstrap in Regression

Once you've fit a regression — say, a factor model regressing stock returns on market and size factors — you often want a bootstrap confidence interval on a coefficient, since the textbook formula for standard errors assumes things (constant error variance, no fat tails) that real financial data routinely violates. But there are two genuinely different ways to bootstrap a regression, and they rest on different assumptions about what's "random" in your data: is it the whole $(x, y)$ pair that's a random draw from some population, or is $x$ fixed and only the leftover error around the fitted line random? Picking the wrong one for your situation gives a confidence interval that's technically computed correctly but answers a question you didn't actually mean to ask.

An analogy: resampling students versus resampling exam-day luck

Imagine a regression of exam scores on hours studied, across a class of students. The pairs bootstrap says: treat each (hours-studied, score) pair as a randomly sampled unit from the population of students, and resample whole students with replacement — as if you picked a fresh class of the same size from the same population. This makes sense if you think of the students themselves as randomly drawn, and both their study habits and scores as part of what varies from sample to sample. The residual bootstrap instead says: treat each student's hours-studied as fixed (as if the study hours were assigned, not random) and instead resample the leftover exam-day luck — the residual between each student's actual score and what the fitted line predicted for their hours — reattaching a randomly resampled residual to each student's fixed hours-studied value to generate a new hypothetical score. This makes sense only if you believe the "luck" component is drawn from the same distribution for every student, regardless of how many hours they studied.

The methods, one symbol at a time

Given a fitted regression $y_i = x_i^\top \hat\beta + \hat{e}_i$ for $i = 1, \ldots, n$, where $\hat{e}_i$ is the observed residual:

Pairs bootstrap: draw $n$ observations with replacement from the original $\{(x_i, y_i)\}$ pairs (resampling whole rows), refit the regression on each resampled dataset to get $\hat\beta^{*}$, and repeat many times to build up the sampling distribution of $\hat\beta^{*}$. This makes no assumption about the errors' distribution being the same across observations — it treats the whole joint $(x,y)$ relationship as what's random.

Residual bootstrap: keep the original $x_i$ values fixed across every resample, draw resampled residuals $\hat{e}_i^{*}$ with replacement from the pool of observed residuals $\{\hat{e}_1, \ldots, \hat{e}_n\}$, construct new synthetic outcomes $y_i^{*} = x_i^\top \hat\beta + \hat{e}_i^{*}$, and refit the regression on $(x_i, y_i^{*})$ to get $\hat\beta^{*}$. This assumes the residuals are exchangeable — drawn from a common distribution independent of $x_i$ — which fails if the error variance changes with $x_i$ (heteroskedasticity), a very common feature of financial data where, for instance, high-beta stocks tend to have noisier residuals than low-beta stocks.

Worked example: a two-factor return regression

Suppose you regress a stock's monthly excess return on market beta and a size factor over 120 months, getting $\hat\beta_{market} = 1.2$, $\hat\beta_{size} = 0.4$. Using the pairs bootstrap, you resample 120 months (with replacement) treating each month's full triple of return, market factor, and size factor as one resampled unit; if some months have both unusually large factor exposures and unusually large residuals together (say, crisis months where everything moves together), the pairs bootstrap naturally preserves that joint behavior, since it resamples the whole observed relationship as-is. Using the residual bootstrap, you'd keep every month's market and size factor values fixed exactly as observed, and only resample which month's residual gets attached to which month's factor values — this implicitly assumes a crisis month's factor exposures could just as easily have paired with a calm month's small residual, which may not be realistic if residual size and factor exposure are actually related (heteroskedasticity tied to the regime).

The explorer below fits an OLS line and shows its residuals directly; picture the residual spread widening or narrowing across the x-axis, and ask whether it would be safe to pool and reshuffle all those residuals as if they were exchangeable — that question is exactly what decides between the two bootstrap approaches.

Worked example: detecting heteroskedasticity that breaks the residual bootstrap

Suppose you plot the regression's squared residuals against the market factor and see a clear pattern: months with large positive or negative market moves have residuals roughly three times larger in magnitude than calm months. Bootstrapping residuals by pooling all 120 residuals together and reattaching them randomly to any month's factor values will, on average, attach a "calm-month-sized" residual to a "crisis-month" factor value about as often as the reverse — smoothing over the real pattern where crisis months should get bigger residuals. The pairs bootstrap sidesteps this entirely, since it never separates the residual from the month it actually came from; whatever heteroskedasticity pattern exists in the real data is preserved automatically because each resampled unit is a real historical month, not a reconstructed synthetic one.

What this means in practice

Financial regressions — factor models, hedge ratios, event-study coefficients — very often have heteroskedastic residuals (volatility clusters, crisis periods have larger errors across the board), which favors the pairs bootstrap as the safer default, since it makes no assumption about the error structure and simply resamples whatever joint behavior actually occurred historically. The residual bootstrap is preferable when you have good reason to believe $x$ values are genuinely fixed by design (a designed experiment, not observational financial data) and residuals really are exchangeable, since it can produce tighter, more efficient confidence intervals when its assumption holds — the wild bootstrap (see wild bootstrap) is a further refinement of the residual approach specifically built to handle heteroskedasticity without abandoning the residual-resampling framework entirely.