Optimal Ensemble Weights

Once you have several models predicting the same target — a boosted tree model, a random forest, a linear model — the simplest way to combine them is an unweighted average. But some models are more accurate than others, and some models make similar mistakes to each other while others don't, and both facts matter for how to weight them. Finding the mathematically best combination turns out to be structurally identical to a problem quant readers already know well: building a minimum-variance portfolio out of correlated assets.

The error-correlation parallel

Think of each model's prediction error (its residual, the difference between its prediction and the truth) as a "return stream." Combining models means combining these error streams with some set of weights, wanting the combined error to have the smallest possible variance — exactly the objective of a minimum-variance portfolio, where "assets" are models and "returns" are their errors. Just as a portfolio of two highly correlated assets doesn't diversify away much risk, an ensemble of two models that tend to be wrong on the same examples doesn't reduce error much no matter how they're weighted. Conversely, two models with the same accuracy but weakly correlated errors can combine into something meaningfully better than either alone, because their mistakes partially cancel when averaged.

In plain English: an ensemble's benefit comes from its members disagreeing about which examples are hard, not (only) from each member being independently accurate. The optimal weight on each model reflects both its own accuracy and how redundant its errors are with everyone else's.

The formula, restated in words

For $n$ models with error covariance matrix $\Sigma$ (the same object as a return covariance matrix in portfolio theory), the variance-minimizing combination weights are:

$$w^* = \frac{\Sigma^{-1} \mathbf{1}}{\mathbf{1}^\top \Sigma^{-1} \mathbf{1}} ,$$

the same closed-form solution as the minimum-variance portfolio, with $\mathbf{1}$ a vector of ones. In plain English: weight each model inversely by how much unique (uncorrelated) error variance it contributes, exactly as a minimum-variance portfolio underweights assets that are both volatile and redundant with the rest of the portfolio.

Worked example: two models, different correlation regimes

Suppose two models each have prediction error variance of 4 (equally accurate individually). If their errors are highly correlated ($\rho = 0.9$), the minimum-variance combination weights them close to 50/50, and the combined error variance only drops to about 3.8 — barely better than either alone, since their mistakes largely overlap. If instead their errors are weakly correlated ($\rho = 0.1$), the same equal-accuracy models combined at 50/50 bring the combined variance down to about 2.2 — a much larger reduction, because when one model is wrong, the other is often right, and averaging cancels much of the error. The individual accuracy was identical in both scenarios; the entire difference in benefit came from error correlation.

The two-asset frontier explorer above makes the same tradeoff visible for a portfolio: drag the correlation slider and watch how lower correlation between the two "assets" (here, models' errors) lets a combined portfolio (ensemble) reach a lower risk (error) than either asset alone, for the same expected return (accuracy).

What this means in practice

Before blindly averaging model outputs or hand-picking equal weights, check the correlation of their errors on a validation set — if two candidate models are highly correlated, adding the weaker one to the ensemble may add complexity without meaningfully reducing error, while a much less accurate but structurally different model (say, a linear model alongside two tree ensembles) can be worth including precisely because its mistakes are different. In practice, weights are usually estimated by fitting a constrained regression (weights summing to one, often non-negative) on out-of-fold predictions rather than by solving the covariance formula directly, since raw covariance estimates from limited validation data are noisy.