Irreducible Error and the Bayes Rate

A casino's roulette wheel is a machine for generating genuine, irreducible randomness — no amount of studying past spins tells you where the ball lands next, because the outcome depends on physical details too fine and chaotic to ever measure completely. Contrast a poorly calibrated scale that always reads two pounds heavy: that error is fixable, once noticed, by recalibrating. Every prediction problem mixes these two kinds of error — the roulette-wheel kind no model can ever remove, and the miscalibrated-scale kind a better model can. Irreducible error is the roulette-wheel part.

The decomposition

Write the true relationship as $y = f(x) + \varepsilon$, where $f(x)$ is the real, learnable function relating inputs to output and $\varepsilon$ is random noise, unrelated to $x$, with mean zero and variance $\sigma_\varepsilon^2$. For any fitted model $\hat{f}$, the expected squared prediction error at a point $x$ decomposes as

$$\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(f(x) - \mathbb{E}[\hat{f}(x)]\big)^2}_{\text{bias}^2} + \underbrace{\text{Var}(\hat{f}(x))}_{\text{variance}} + \underbrace{\sigma_\varepsilon^2}_{\text{irreducible error}}$$

In words: total expected error splits into three pieces. Bias is how far off the model's average prediction is from the truth — a systematic tendency to miss in the same direction, like the miscalibrated scale. Variance is how much the model's prediction would change if you retrained it on a different sample of the same size — how jumpy it is. Irreducible error, $\sigma_\varepsilon^2$, is the noise term's own variance — it doesn't depend on the model $\hat f$ at all, which is exactly why no choice of model, however good, can shrink it. It is a property of the problem, not the model.

Watch the train and test error curves against model complexity. Both curves flatten toward some floor as complexity increases rather than continuing down to zero — that floor is irreducible error. Bias and variance can be traded against each other by adjusting complexity, which is why the curves move at all; irreducible error is the flat baseline underneath both of them that no amount of sliding along this curve ever removes.

Worked example 1: separating bias, variance, and irreducible error numerically

Suppose at a specific input $x_0$, the true value is $f(x_0) = 10$, and the noise has $\sigma_\varepsilon^2 = 4$ (so realized $y$ values scatter around 10 with variance 4, i.e., a standard deviation of 2). A particular fitted model, averaged over many retrainings on different samples, predicts $\mathbb{E}[\hat f(x_0)] = 9.0$ with $\text{Var}(\hat f(x_0)) = 1.5$.

Bias$^2 = (10 - 9.0)^2 = 1.0$. Variance $= 1.5$. Irreducible error $= 4$. Total expected squared error $= 1.0 + 1.5 + 4 = 6.5$. Even though the model has both bias and variance issues contributing $2.5$ combined, the $4.0$ from irreducible error is the single largest piece and the only piece guaranteed to remain no matter how the model is improved — the absolute best any model could achieve at this point, with zero bias and zero variance, is still an expected squared error of $4.0$, not $0$.

Worked example 2: two "improvements" that behave very differently

A team improves their model twice. Improvement A reduces bias$^2$ from $1.0$ to $0.2$ by adding a missing feature that genuinely explains part of $f(x)$ — total error drops from $6.5$ to $5.7$. Improvement B squeezes out more accuracy by adding twenty engineered features and a larger model; it drives training error down substantially, but held-out variance rises from $1.5$ to $3.0$ from overfitting sample-specific noise — total held-out error becomes $7.2$, worse than before, despite the training loss looking better than ever.

Neither improvement touched the $4.0$ irreducible floor — it's identical in both scenarios, exactly as the theory says it must be. But the difference between A and B — a real gain versus a real regression — is invisible if you only watch training loss, and only becomes visible by decomposing held-out error and noticing which of bias or variance is moving.

What this means in practice

Estimating the irreducible error floor — even roughly — tells a quant team when to stop trying to improve a model and start accepting the result. If held-out error has been driven down to somewhere near a plausible noise floor for daily stock returns (which, given how close markets are to efficient, is large relative to any real signal), continued feature engineering is very likely just trading bias for variance back and forth rather than closing a real gap, and the marginal effort is better spent finding a different signal entirely rather than squeezing the current model harder.