Ridge regression, paying bias to buy variance

Question

Ordinary least squares is unbiased, yet practitioners routinely shrink coefficients toward zero with ridge regression, which minimizes $\sum_i (y_i - x_i^\top\beta)^2 + \lambda\|\beta\|^2$ .

Explain, in bias–variance terms, why intentionally biasing the estimates can reduce prediction error, and describe how the penalty $\lambda$ controls the trade.

Accepted Answer

What ridge does. The penalty \lambda\beta\^2 pulls every coefficient toward zero. In the clean orthonormal case each ridge coefficient is just the OLS one scaled down: \hat\beta^{	ext{ridge}}j = \frac{\hat\beta^{	ext{OLS}}j}{1 + \lambda}. So ridge estimates are systematically too small in magnitude, they are biased. Why bias can help. Prediction mean squared error decomposes as 	ext{bias}^2 + 	ext{variance} + 	ext{irreducible noise}. OLS puts all its weight on zero bias, but when predictors are many or collinear, its variance explodes, small data wiggles swing the coefficients wildly. S…

Ridge regression, paying bias to buy variance

Your answer

More Statistics questions