Ensemble Diversity and Model Correlation

Averaging several models together is the standard way ensembles reduce variance. But averaging only helps to the degree the models actually disagree in their errors. If ten models trained the same way on the same data make nearly identical mistakes, averaging them barely moves the needle — you've spent ten times the computation for almost no benefit. Diversity, not just each member's individual accuracy, is what determines how much an ensemble actually gains.

The analogy: ten thermometers versus one copied ten times

Ten genuinely independent thermometers, each with its own small unrelated error, average those errors down substantially, since errors tend to cancel. But copy one thermometer's reading ten times and "averaging" gives back the exact same reading, error and all — no improvement, because there was never any independent information to average across. Ensemble diversity is the difference between ten real thermometers and one copied ten times.

The mechanics: variance reduction scales with correlation, not just count

For $M$ models with individual variance $\sigma^2$ and average pairwise error correlation $\rho$, the ensemble average's variance is:

$$\text{Var}(\bar f) = \rho \sigma^2 + \frac{1-\rho}{M}\sigma^2 .$$

In plain English: as $M \to \infty$ the second term vanishes, but $\rho\sigma^2$ never goes away — a floor set entirely by correlation, no matter how many models you add. At $\rho=1$ (identical errors) averaging buys nothing; at $\rho=0$ variance shrinks toward zero. Bagging's random resampling and random forests' random feature subsets exist specifically to push $\rho$ down.

Worked example: computing the variance floor

Ten models, each $\sigma^2=4$, $\rho=0.5$: ensemble variance is $0.5(4) + \frac{0.5}{10}(4) = 2.2$ — modest improvement over 4, nowhere near the $0.4$ you'd get at $\rho=0$. Dropping correlation to $\rho=0.1$ (say, via random feature subsampling) gives $0.1(4) + \frac{0.9}{10}(4) = 0.76$ — nearly fivefold better than the $\rho=0.5$ case, using the exact same ten base models, purely from making their errors less alike.

Drag the exponent above: a small drop in $\rho$ has an outsized effect on the variance floor, much like a small change in exponent reshapes how steeply the curve falls — correlation is the lever that matters most, not model count.

What this means in practice

Individually strong but highly correlated trading signals add much less value than their count suggests — ten momentum variants that fail during the same regime shifts behave, in aggregate, like one. Actively seeking diversity — different feature subsets, different windows, structurally different model families — usually helps more than squeezing extra accuracy from a model that already resembles the rest of the ensemble.