Underfit, balanced, or overfit: pick the best

Question

You are picking among three models of increasing complexity. Expected test error is $\text{squared bias} + \text{variance} + \text{irreducible noise}$ , and the noise is 2 for all three:

Simple model: squared bias 16, variance 1
Medium model: squared bias 4, variance 3
Complex model: squared bias 0, variance 9

Which model has the lowest expected test error, and what is it?

Accepted Answer

The idea. As a model gets more flexible, its bias falls (it can fit more shapes) but its variance rises (it chases noise). The irreducible term is fixed. Add all three and compare. Total the pieces (noise 2 throughout): - Simple: 16 + 1 + 2 = 19 - Medium: 4 + 3 + 2 = 9 - Complex: 0 + 9 + 2 = 11 The winner. The medium model, with test error 9. The simple model is badly underfit (bias dominates), the complex model is overfit (variance dominates), and the medium model balances them. This U-shape, error high at both ends and lowest in the middle, is the signature of the bias-variance tradeoff. It…

Underfit, balanced, or overfit: pick the best

Your answer

More Statistics questions