Why Ensembles Work: The Ambiguity Decomposition

At a county fair, a jar holds a large number of jellybeans, and hundreds of visitors each guess how many there are. Almost every individual guess is bad — some wildly too high, some wildly too low. And yet the average of all those guesses is consistently close to the true count, often closer than any single expert's guess. That fact is not a cute anecdote. For a certain, very common kind of error, it is a mathematical identity — the average of a set of forecasts is provably at least as accurate as the average forecast's own accuracy, and the amount by which it is better is an exact, computable number.

The analogy: doctors disagreeing on a diagnosis

Ask three independent doctors to estimate a patient's risk score from an ambiguous set of symptoms and you will get three different numbers, each shaped by that doctor's particular experience and blind spots. If the three doctors made the same mistake — sharing a common blind spot — averaging their answers keeps that mistake. But if their mistakes point in different directions — one overestimates for reasons unrelated to why another underestimates — averaging cancels those individual errors out, the way opposite-signed numbers partially cancel in a sum. The jellybean jar works because different guessers' errors come from different, largely unrelated causes. An ensemble of models works for exactly the same reason: what matters is not how good each individual model is, but how differently wrong they are from each other.

The exact formula

Consider $m$ models each predicting a number, call the predictions $f_1, f_2, \ldots, f_m$, with true value $y$. Their simple average is $\bar f = \frac{1}{m}\sum_i f_i$ — this average is itself the "ensemble" prediction. For squared error, there is an exact algebraic identity, no approximation involved:

$$(\bar f - y)^2 = \underbrace{\frac{1}{m}\sum_i (f_i - y)^2}_{\text{average individual error}} \;-\; \underbrace{\frac{1}{m}\sum_i (f_i - \bar f)^2}_{\text{average ambiguity}}$$

Read the left side first: the ensemble's own squared error. The right side splits it into two pieces. Average individual error is exactly what it sounds like — take every single model's squared error against the truth, and average them, ignoring the ensemble entirely. Average ambiguity measures how much the individual predictions disagree with each other — specifically, with the ensemble average itself — regardless of whether any of them are close to the truth. Because ambiguity is a sum of squares, it can never be negative, which means the equation guarantees something powerful on its own: the ensemble's error can never exceed the average individual error, and it falls short of it by exactly the amount the models disagreed. Models that are individually mediocre but disagree a lot can still produce a genuinely accurate ensemble; models that are individually good but always agree gain almost nothing from being combined.

Worked example 1: three models, moderate disagreement

Three models predict a value whose truth is $y = 10$: predictions $8$, $11$, $13$.

$$\bar f = \frac{8+11+13}{3} = 10.667$$

Average individual error:

$$\frac{(8-10)^2 + (11-10)^2 + (13-10)^2}{3} = \frac{4+1+9}{3} = \frac{14}{3} = 4.667$$

Average ambiguity (each prediction's squared distance from the ensemble average, $10.667$):

$$\frac{(8-10.667)^2 + (11-10.667)^2 + (13-10.667)^2}{3} = \frac{7.111 + 0.111 + 5.444}{3} = \frac{12.667}{3} = 4.222$$ $$\text{ensemble error} = 4.667 - 4.222 = 0.445$$

Check directly against the ensemble prediction: $(\bar f - y)^2 = (10.667 - 10)^2 = 0.445$. It matches exactly — this is an identity, not an estimate. Notice the average individual model was off by $4.667$ in squared-error terms, roughly ten times worse than the ensemble's $0.445$, purely because the three models' errors pointed in different, partially offsetting directions.

Worked example 2: perfect cancellation

Predictions $6$, $10$, $14$ against the same truth $y = 10$.

Average individual error: $\frac{(6-10)^2 + (10-10)^2 + (14-10)^2}{3} = \frac{16+0+16}{3} = 10.667$

Ensemble average: $\bar f = \frac{6+10+14}{3} = 10$, exactly the truth.

Average ambiguity: $\frac{(6-10)^2 + (10-10)^2 + (14-10)^2}{3} = \frac{16+0+16}{3} = 10.667$

$$\text{ensemble error} = 10.667 - 10.667 = 0$$

Two of the three models were off by 4 in either direction, individually contributing a large squared error each — yet the ensemble is perfect, because those errors were symmetric around the truth and canceled exactly. The formula shows this is not a coincidence: whenever ambiguity happens to equal average individual error, the ensemble error is driven to zero, regardless of how bad any single model looked on its own.

The bias-variance explorer above is directly relevant here: the ambiguity term is a variance-type quantity — spread among predictions around their own average — while the average individual error contains both bias and variance. Ensembling attacks the variance piece, exactly as The Bias-Variance Decomposition describes for averaging independent copies of one model; the ambiguity decomposition is that same mechanism proven exactly, for any collection of different models, not just resampled copies of an identical one.

What this means in practice

The identity explains precisely why ensemble methods in finance — random forests, or blending several independently developed alpha models — are built to maximize diversity, not just individual accuracy. Two models trained on the same features with the same algorithm and nearly the same hyperparameters will agree with each other almost everywhere, driving the ambiguity term toward zero and leaving the ensemble barely better than either alone. A forest built from decision trees, each trained on a different bootstrap sample and a different random subset of features, deliberately manufactures disagreement, and the ambiguity term is the mathematical reason that manufactured disagreement pays off. This is also why simply adding more copies of a highly correlated model to a portfolio-level ensemble of strategies helps far less than a manager might expect — correlated errors do not cancel.

Practice

1. Five models all predict exactly the same value, which happens to differ from the truth. What does the ambiguity term equal, and what does that tell you about how much the ensemble helps?
2. Using the ambiguity decomposition, explain in one sentence why a forest of highly correlated trees underperforms a forest of decorrelated trees, even if both forests have the same average per-tree accuracy.
3. Three models predict 20, 22, 24 for a true value of 24. Compute the average individual error, the average ambiguity, and the ensemble error, and check that the identity holds.