Which Learners Benefit From Bagging

Bagging trains the same algorithm on many bootstrap resamples and averages the results, hoping to cancel out each fit's noise. This only helps if individual fits actually have different noise to cancel — that is, if the learner is unstable, meaning its output changes noticeably when the training data is perturbed slightly. Some learners are naturally unstable and gain a lot; others are stable and gain almost nothing, because bagging ten resample-fits of a stable learner gives back ten nearly identical answers.

The analogy: a wobbly table leg versus a solid one

Bagging is like averaging several attempts to measure something with a wobbly instrument — if it genuinely wobbles differently each time, averaging smooths that out. But a rock-solid instrument gives essentially the same reading regardless of small changes; averaging several such readings gives back the same number a single reading would, since there was no wobble to average away.

The mechanics: instability is what bagging exploits

A learner's instability is how much its fitted function changes when a training point is added, removed, or reweighted, as happens across bootstrap resamples. A fully-grown decision tree is a textbook unstable learner: one changed point near the top can flip which feature gets the first split, cascading into a completely different tree below. Linear regression is stable: removing one point out of many nudges coefficients only slightly, since each is a smooth function of all the data, not a discrete decision that can flip. Bagging's variance reduction, per the ensemble correlation formula, scales with how much fits actually differ — and resampling only creates that difference if the learner is sensitive enough to notice it.

$$\text{Var}(\bar f_{\text{bagged}}) = \rho \sigma^2 + \frac{1-\rho}{M}\sigma^2 .$$

In plain English: for a stable learner, $\rho \approx 1$ (resamples barely change the fit) and the formula collapses to roughly $\sigma^2$ — no gain. For an unstable learner, $\rho$ sits meaningfully below 1, and averaging genuinely reduces variance.

Worked example: bagging a tree versus bagging a linear model

Bagging a decision stump and a linear regression, both predicting earnings growth from ten ratios, across 50 bootstrap resamples of 200 companies: the stump's chosen splitting feature shifts noticeably resample to resample — sometimes leverage, sometimes revenue growth — because a few companies entering or leaving the sample tip the balance. This instability means the 50 stumps genuinely disagree, and averaging measurably reduces variance. The linear regression's ten coefficients barely move across resamples — dropping a handful of companies out of 200 shifts each by a fraction of a standard error — so the 50 bagged fits are nearly identical, and averaging provides almost no improvement.

The high-complexity end of the curve above — where test error is most volatile — is exactly where unstable learners like deep trees live, and where bagging's variance reduction does the most work.

What this means in practice

Bagging is worth applying to high-variance, low-bias learners — deep decision trees especially, which is why random forests (bagged trees plus extra decorrelation) are effective. It's largely wasted on already-stable learners like linear regression, ridge, or k-NN with large $k$ — bagging adds computation with little change, and any perceived backtest improvement is more likely noise.