The Bias-Variance Decomposition

Two models can post the same disappointing test error for completely different reasons, and knowing which reason applies changes what you should do next. One model might be making the same mistake every time — consistently underestimating a stock's fair value, say — which more data will never fix, because the pattern in the data was never rich enough for the model to capture. The other might be making a different mistake every time it is retrained, wildly over- and under-shooting depending on which historical days happened to land in the training set, which is a problem more data or a simpler model can genuinely fix. The bias-variance decomposition is the formula that tells these two failure modes apart.

The analogy: two ways to miss the bullseye

Imagine two archers, each shooting five arrows at a target. The first archer's arrows land in a tight cluster, but the cluster is a foot to the left of the bullseye — every shot is consistent, and every shot is wrong in the same way. That archer is biased: fixing it means changing their aim, not their steadiness. The second archer's arrows scatter widely around the bullseye — some land dead center, some land a foot away in every direction — but if you averaged all five arrows' positions, that average would sit right on the bullseye. That archer is not biased; they are high variance: fixing it means steadying their hand, not changing their aim.

A model trained repeatedly on different samples of the same market is exactly this archer. Each retraining is one arrow. If every retrained version makes the same systematic error, that is bias. If different retrained versions swing wildly in different directions, that is variance. Total error is driven by both, and reducing one does not automatically reduce the other — a steadier archer aiming at the wrong spot still misses.

The formula

For a prediction target $y$ with true underlying function $f(x)$ plus unavoidable noise $\varepsilon$ (so $y = f(x) + \varepsilon$), and a model $\hat f(x)$ trained on one particular random sample of data, the expected squared error at a point $x$, averaged over every possible training sample the model could have been handed, decomposes as

$$\mathbb{E}\big[(y - \hat f(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat f(x)] - f(x)\big)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{irreducible noise}}$$

In words, reading each term separately: bias is how far the average prediction, across every possible training sample you could have drawn, sits from the true value — the archer's average arrow position minus the bullseye. Variance is how much any single prediction typically swings around that average — how scattered the arrows are around their own cluster center, regardless of where that center is. Irreducible noise is randomness baked into the target itself — the wind gusting during every shot — which no model, however good, can remove. The three terms add up to the total expected squared error, and improving one does not automatically improve the others.

Worked example 1: two models, same target, different failure

The true value at some point is $f(x) = 5.0$. Two models are each retrained five times on five different resampled datasets, and each retrained version's prediction at $x$ is recorded.

Model A (a simple, heavily smoothed model) predicts: 4.0, 4.1, 3.9, 4.0, 4.0.

$$\text{mean} = \frac{4.0+4.1+3.9+4.0+4.0}{5} = 4.00, \qquad \text{bias} = 4.00 - 5.00 = -1.00, \qquad \text{bias}^2 = 1.00$$ $$\text{variance} = \frac{0^2 + 0.1^2 + (-0.1)^2 + 0^2 + 0^2}{5} = \frac{0.02}{5} = 0.004$$

Model B (a flexible, lightly regularized model) predicts: 5.5, 3.0, 6.2, 4.0, 6.3.

$$\text{mean} = \frac{5.5+3.0+6.2+4.0+6.3}{5} = 5.00, \qquad \text{bias} = 5.00 - 5.00 = 0$$ $$\text{variance} = \frac{0.5^2 + (-2.0)^2 + 1.2^2 + (-1.0)^2 + 1.3^2}{5} = \frac{0.25+4.00+1.44+1.00+1.69}{5} = \frac{8.38}{5} = 1.676$$

With irreducible noise $\sigma^2 = 0.05$ for both, total expected squared error is $1.00 + 0.004 + 0.05 = 1.054$ for Model A versus $0 + 1.676 + 0.05 = 1.726$ for Model B. Model B is unbiased on average and Model A is badly biased, yet Model A wins decisively, because its variance is negligible while Model B's is enormous. Being right on average across many retrainings is worthless if any single retraining is wildly unpredictable — you only ever get to deploy one trained model, not the average of infinitely many.

Worked example 2: why averaging fixes variance but not bias

Model B's variance of 1.676 came from a single trained model. Suppose instead two independently retrained copies of Model B are averaged together at prediction time. For independent estimators, the variance of an average of $m$ copies is the single-copy variance divided by $m$:

$$\text{Var}\left(\frac{\hat f_1(x) + \hat f_2(x)}{2}\right) = \frac{\text{Var}(\hat f(x))}{2} = \frac{1.676}{2} = 0.838$$

Averaging two copies cut variance from 1.676 to 0.838 — nearly in half, exactly as the formula predicts — without touching the bias, which was already zero and stays zero, because averaging unbiased things is still unbiased. This is precisely the mechanism behind ensembling: it attacks variance, not bias, which is why an ensemble of independently biased models (all consistently wrong the same way) does not help nearly as much as an ensemble of independently high-variance ones.

Drag the complexity slider above and watch the two curves. Low complexity is the left side of the plot — high bias, near-zero variance, like Model A. High complexity is the right side — near-zero bias, high variance, like Model B. The total error curve (their sum, plus the irreducible floor) dips somewhere in the middle, which is exactly what the two worked examples above computed by hand at two single points on that curve.

What this means in practice

Simple, heavily regularized models (a linear regression on a handful of macro factors) sit on the high-bias, low-variance side: retrain them on a different five-year window and the coefficients barely move, but they may never capture a real nonlinear effect that is genuinely there. Flexible models (a deep gradient-boosted tree on thousands of features) sit on the low-bias, high-variance side: they can capture real nonlinear structure, but retrain on a slightly different window and the predictions can shift substantially. The right choice depends on how much data you actually have relative to how complex the true relationship is — more data shrinks variance for a fixed model, which is why the same flexible model that overfits on two years of daily data can generalize fine on twenty.

Practice

1. A model retrained on ten different bootstrap samples produces predictions at a fixed point that are all within 0.01 of each other but consistently 2.0 away from the true value. Which term in the decomposition dominates its error?
2. Averaging four independent copies of a model with variance 0.80 and zero bias — what is the resulting variance, and what is the resulting bias?
3. Why can't irreducible noise be reduced by using a better model, more data, or more regularization?