Why does ridge regression fix multicollinearity?

Question

Ridge regression is the standard fix when regressors are highly correlated.

Explain how ridge reduces the variance, why it introduces bias, and how it acts specifically on the ill-determined direction.

Accepted Answer

The mechanism. OLS solves X'X\,\beta = X'y; ridge solves (X'X + \lambda I)\,\beta = X'y. Near-collinearity makes X'X nearly singular: along the direction where the correlated signals differ (their "twin" axis), the data has almost no variation, giving X'X a tiny eigenvalue. Coefficient variance scales like 1/(	ext{eigenvalue}), so that direction is exactly where variance explodes. Adding \lambda I lifts every eigenvalue by \lambda, which barely touches the large, well-determined eigenvalues but dramatically stabilizes the tiny ones. So ridge shrinks the estimate most along the ill-determined…

Why does ridge regression fix multicollinearity?

Your answer

More Statistics questions