Bagging and Variance Reduction

A single deep decision tree, trained on one dataset, is wildly sensitive to which exact rows happened to be in the training set — swap a handful of data points and the whole tree can change shape. That sensitivity is variance, and a high-variance model that's accurate on average can still be unreliable on any given run. Bagging is a way to keep the model's flexibility while taming that instability, without touching the model itself.

The analogy: averaging many noisy rulers

Suppose you want to measure a table's length but your only ruler is slightly warped, giving a reading that's off by a random amount each time you use it. One measurement is unreliable. But if you measure ten times, each time picking the ruler up and putting it down slightly differently, and average the ten readings, the random wobbles mostly cancel out — the errors that pushed some readings too high are offset by errors that pushed others too low. The table's true length hasn't changed and neither has any individual ruler's tendency to wobble; only the average has gotten more reliable. Bagging does this with models instead of rulers.

The maths: bootstrap, then average

"Bagging" is short for bootstrap aggregating. Given a training set of $n$ points, draw $B$ new datasets, each of size $n$, by sampling with replacement from the original — this is the bootstrap. Train one model on each resampled dataset, then combine:

$$\hat{f}_{\text{bag}}(x) = \frac{1}{B}\sum_{b=1}^{B} \hat{f}_b(x)$$

In words: build $B$ separate models, each seeing a slightly different random resample of the same data, and average their predictions (for classification, take a majority vote instead). Sampling with replacement means some original rows appear multiple times in a given resample and others don't appear at all — each resample is a plausible "parallel universe" version of the same dataset. The variance reduction comes from a basic fact of averaging: if $B$ estimates each have variance $\sigma^2$ and are only partially correlated with correlation $\rho$, their average has variance

$$\text{Var}(\hat f_{\text{bag}}) = \rho\sigma^2 + \frac{1-\rho}{B}\sigma^2$$

In words: the second term shrinks toward zero as $B$ grows, but the first term, driven by how correlated the individual models are, doesn't. This is why bagging works best on models that vary a lot between resamples (like deep trees) and why random forests go a step further by decorrelating the trees themselves.

Worked example 1: averaging cuts the variance in half

Suppose each of five bagged models has prediction variance $\sigma^2 = 4$ on a given point, and their pairwise correlation is $\rho = 0.5$. Plugging into the formula: $\text{Var} = 0.5(4) + \frac{0.5}{5}(4) = 2 + 0.4 = 2.4$. A single model's variance was 4; the bagged average's variance is 2.4 — a real reduction, but capped by the $\rho\sigma^2 = 2$ floor no amount of extra averaging can get below, since the models are correlated (they're trained on overlapping resamples of the same data).

Worked example 2: a toy resample by hand

Original dataset: five house prices, $\{200, 210, 400, 205, 215\}$ (in thousands), where $400$ is an unusual outlier. A single bootstrap resample, drawn with replacement, might be $\{210, 400, 400, 205, 200\}$ — note $400$ appeared twice and $215$ didn't appear at all. A model trained on this resample would be dragged even further toward the outlier than the original data warranted. But a different resample, $\{200, 210, 205, 215, 205\}$, might miss the outlier entirely. Average predictions across many such resamples, and the outlier's influence — present in roughly $1-(1-1/5)^5 \approx 67\%$ of resamples but never doubled every time — washes out toward its fair share rather than dominating every single model.

Drag model complexity in the explorer above: bagging is one of the few tools that reduces variance (the gap between train and test error at high complexity) without simplifying the model, letting you keep a flexible, low-bias model while paying less of the usual variance penalty.

The explorer above shows an estimate settling as more samples are averaged — bagging's variance reduction across $B$ models follows the same shrinking-noise logic, just averaging model predictions instead of raw observations.

What this means in practice

Bagging is most valuable for high-variance, low-bias models — deep decision trees especially — where it converts an unstable, overfit-prone learner into a stable one at the cost of training many models instead of one. It does almost nothing for a model that's already low-variance, like plain linear regression, since there's little instability left to average away.