Deep Ensembles for Uncertainty Estimation

A single trained neural network hands back one prediction and, at best, one confidence number attached to it. But that confidence number was learned from the same training run that produced the prediction — it has no independent way of flagging "actually, I've barely seen data like this." Retrain the identical architecture from a different random weight initialisation and you get a different model, often with a similar overall accuracy but a different opinion on any particular unusual input. That disagreement is information the single model threw away.

The analogy: five analysts, one dataset, five starting assumptions

Give five equally competent analysts the exact same historical dataset but let each build their model independently — different starting assumptions, different order in which they process the data, different arbitrary tie-breaks along the way. On a bread-and-butter question, all five will converge on essentially the same answer, because the data pins it down tightly. On an unusual question the data barely covers, their independent starting points send them off in different directions, and they'll disagree. You didn't have to tell any of them "flag your uncertainty" — the spread of their five answers, measured after the fact, does that for you.

Building the ensemble

Train $M$ networks with identical architecture on the same training set, differing only in random weight initialisation (and optionally the order data is shuffled in). For a regression target, each network $m$ outputs a prediction $\hat{y}_m(x)$. The ensemble's summary prediction and its epistemic-uncertainty estimate are:

$$\bar{y}(x) = \frac{1}{M}\sum_{m=1}^{M} \hat{y}_m(x), \qquad s^2(x) = \frac{1}{M}\sum_{m=1}^{M} \big(\hat{y}_m(x) - \bar{y}(x)\big)^2$$

In words: average the $M$ networks' answers to get one prediction, and measure how far each one strays from that average to get a variance. This $s^2(x)$ is not the network's built-in noise estimate — it is model disagreement, which is exactly the epistemic half of the uncertainty split described in Aleatoric vs Epistemic Uncertainty. If every network is trained to output both a mean and its own aleatoric variance $\sigma_m^2(x)$ (a common design for regression heads), the two sources combine into one predictive variance: $\frac{1}{M}\sum_m \sigma_m^2(x) + s^2(x)$ — average noise plus disagreement.

Worked example 1: five one-unit models on a small dataset, by hand

Simplify to five toy linear models, each fit on a slightly different bootstrap resample of four training points, asked to predict at a new input $x = 2$. Their predictions: $\hat{y} = 4.1, 4.3, 3.9, 4.2, 4.0$.

Mean: $\bar{y} = (4.1+4.3+3.9+4.2+4.0)/5 = 20.5/5 = 4.10$.

Variance: deviations are $0.0, 0.2, -0.2, 0.1, -0.1$; squared, $0.00, 0.04, 0.04, 0.01, 0.01$; average $= 0.10/5 = 0.02$, so $s = 0.14$.

Now ask the same five models to predict at $x = 50$, far outside the training range they ever saw. Predictions diverge sharply: $\hat{y} = 12.0, 41.0, -8.0, 60.0, 5.0$. Mean $= 22.0$; deviations $-10, 19, -30, 38, -17$; squared $100, 361, 900, 1444, 289$; average $= 3094/5 = 618.8$, so $s \approx 24.9$. The prediction spread exploded by a factor of roughly 180, without any of the five models ever being told "you're extrapolating." Disagreement did the flagging automatically.

Worked example 2: sizing a position by ensemble spread

A five-network ensemble predicts next-day return for two stocks, each network's output in percent: Stock A gives $\{0.42, 0.39, 0.44, 0.41, 0.40\}$; Stock B gives $\{0.55, 0.10, 0.90, -0.20, 0.65\}$.

Stock A: mean $= (0.42+0.39+0.44+0.41+0.40)/5 = 2.06/5 = 0.412$, deviations roughly $\pm0.02$, so $s \approx 0.017$ — tight agreement.

Stock B: mean $= (0.55+0.10+0.90-0.20+0.65)/5 = 2.00/5 = 0.400$ — nearly the same central prediction as A — but deviations of $0.15, -0.30, 0.50, -0.60, 0.25$ give squared values $0.0225, 0.09, 0.25, 0.36, 0.0625$, averaging $0.785/5 = 0.157$, so $s \approx 0.396$, over twenty times wider. Two stocks with an almost identical point forecast warrant very different position sizes: a risk system that only looks at the mean prediction would treat them identically and would be wrong to.

What this means in practice

Deep ensembles are popular precisely because they need no change to how each individual network is trained — no special Bayesian machinery, no approximate posterior over weights, just ordinary training repeated $M$ times (typically 5 to 10 is enough to see the disagreement pattern clearly) with different seeds. The cost is real: $M\times$ the training compute and $M\times$ the inference compute, which matters if predictions need to run in a low-latency trading loop. Cheaper approximations exist — Monte Carlo dropout keeps dropout active at inference time and treats repeated stochastic forward passes through one trained network as a crude stand-in for an ensemble — but they tend to underestimate disagreement compared to networks that were genuinely trained separately, because dropout perturbs a single set of learned weights rather than exploring genuinely different solutions the way independent training runs do. In a live system, ensemble spread is a natural gate: route wide-disagreement predictions to a smaller position size, a human reviewer, or simply "no trade," while narrow-disagreement predictions are traded at full conviction.

There's also a secondary benefit that has nothing to do with uncertainty directly: averaging several independently trained networks' predictions is itself a strong accuracy improvement, for the same reason averaging reduces variance in any noisy estimator. A trading desk that adopts deep ensembles purely to get an uncertainty signal often finds the averaged point forecast alone is worth the extra compute, which softens the cost argument against running $M$ models instead of one.

Drag the model complexity here and notice how a single high-complexity model can chase the training data too hard — an ensemble of such models, each overfitting differently, is exactly what produces useful disagreement on new inputs even while each individual model looks confident on its own.