Ridge Regression: Being Deliberately Wrong to Be More Right

For most of the twentieth century, statistics had a sacred cow, and its name was unbiasedness.

The Gauss-Markov theorem, one of the crown jewels of the field, proves that among all linear unbiased estimators, ordinary least squares has the smallest variance. It is the Best Linear Unbiased Estimator. This was taught as a kind of final word. Why would you ever use anything else?

In 1970, two statisticians working in industry, Arthur Hoerl and Robert Kennard, published a paper that pointed at the load-bearing word in that theorem and asked an impertinent question.

Why are we restricting ourselves to unbiased estimators in the first place?

The problem: OLS is the best of a bad crowd

An unbiased estimator is one that is right on average. Run the experiment infinitely many times and the average of your estimates lands exactly on the truth. That sounds like an unambiguously good property.

But notice what it does not promise. It says nothing about how far any individual estimate is from the truth. An estimator can be unbiased and still be catastrophically noisy, throwing out wild estimates that are enormously too high and enormously too low, which happen to average out to the right answer. You do not get to run the experiment infinitely many times. You get one dataset. And "on average, over infinitely many parallel universes, I would be right" is cold comfort when the estimate you actually got is nonsense.

And there is a specific, common situation where ordinary least squares becomes exactly that noisy: when your predictors are highly correlated with each other. Statisticians call this multicollinearity, and the paper's title calls it "nonorthogonal," which is the same idea in geometric language.

Suppose you regress returns on two predictors that are almost identical, two slightly different measures of a company's cheapness, say. The regression has an impossible task: it must decide how much credit each one deserves, and the data cannot tell them apart. So it becomes desperate. It will assign an enormous positive coefficient to one and an enormous negative coefficient to the other, which happen to cancel. Add three more months of data and the two coefficients might swap places entirely.

The estimates are still unbiased. They are also completely useless. This is the situation Hoerl and Kennard set out to fix, and it is the situation every quant lives in, because financial predictors are almost never independent.

The key idea via analogy: a small lie for a big gain

Here is what an estimator's total error is really made of. It has two components:

• Bias: how far off you are systematically.
• Variance: how much you bounce around.

Total error is a combination of both. Ordinary least squares drives bias to exactly zero, and then has no remaining freedom to do anything about variance. In the correlated-predictor case, the variance is enormous. So the total error is enormous, even though the bias is perfect.

Hoerl and Kennard's move: accept a small amount of bias in exchange for a large reduction in variance. If the trade is good, and they proved it can be, your total error goes down. You are systematically slightly wrong, and yet closer to the truth than the estimator that is right on average.

The mechanism is disarmingly simple. Ordinary least squares says: choose the coefficients that minimize the squared errors. Ridge regression says: choose the coefficients that minimize the squared errors plus a penalty proportional to the sum of the squared coefficients.

That penalty is a tax on being large. It makes the regression reluctant to hand out huge coefficients. And that reluctance is exactly what kills the pathological "plus a million, minus a million" solution, because the penalty on those two enormous coefficients is astronomically expensive. The regression is forced to find a sensible, modest, stable compromise: give both correlated predictors a moderate share of the credit.

The analogy: ordinary least squares is a witness who insists on reporting exactly what they saw, including all the noise, hallucinations and misperceptions, because they refuse on principle to shade the truth. Ridge regression is a witness who is willing to round off and be a little conservative, and who therefore ends up giving a far more useful account.

There is a beautiful, and famous, result at the heart of this. Hoerl and Kennard proved that there always exists some amount of penalty, greater than zero, that produces a lower total error than ordinary least squares. Not "sometimes." Always. The unbiased estimator is never the best you can do. That is a genuinely shocking theorem, and it took the field a long time to accept it.

The name comes from the "ridge trace," a plot the authors used to show how the coefficients evolve as you crank the penalty up from zero.

Why it mattered

• It broke the tyranny of unbiasedness. This is the philosophical payload. After Hoerl and Kennard, and after Stein's related work on shrinkage in the 1950s, statisticians could no longer treat bias as an automatic disqualification. Deliberately biased estimators became respectable, then standard, then dominant.
• It is the ancestor of everything. Every regularized model in modern machine learning descends from this idea. The lasso is ridge with a different penalty. The elastic net is a blend of both. Weight decay in a neural network is ridge regression, under a different name. Gradient boosting is riddled with shrinkage. This 1970 paper on an industrial statistics problem is the root of the tree.
• It is the right default in finance. Financial predictors are correlated, data is limited, and noise is overwhelming. That is precisely the regime where ordinary least squares fails and ridge shines. Gu, Kelly and Xiu include penalized linear models as core contenders in their machine learning horse race for stock returns for exactly this reason.
• It shows up in portfolio construction too. Shrinking a noisy covariance matrix toward a stable target (as in Ledoit and Wolf's work) is the same idea, applied to a different object. So is shrinking noisy expected return estimates. The whole discipline of taming estimation error in portfolio optimization is ridge thinking.

The honest limitations

• It never gives you a zero. Ridge shrinks every coefficient toward zero but pins none of them to zero. All two hundred of your useless predictors remain in the model with tiny, non-zero weights. You get stability but not a short list, and no interpretability gain. This is precisely the gap the lasso was invented to fill.
• Choosing the penalty is the whole game, and there is no formula. Too little penalty and you are back to unstable OLS. Too much and you have crushed the real signal along with the noise, leaving a model that predicts nothing but the average. Hoerl and Kennard's own proposals for choosing it were somewhat ad hoc. The modern answer is cross-validation, which in financial time series must respect the arrow of time or you have simply moved your overfitting somewhere you cannot see it.
• The penalty depends on the scale of your variables. A predictor measured in basis points and one measured in percent will be penalised completely differently, which is absurd. You have to standardize your features first. Forgetting to do so is one of the most common and least noticed bugs in applied work.
• The bias is real, and it is in a known direction. Your coefficients are systematically too small. If your purpose is prediction, who cares. If your purpose is to say "a one unit increase in this factor causes a return of exactly X," ridge will lie to you, consistently, and understate the effect.
• It is still a straight line. Ridge fixes the instability of linear regression. It does not make the model any less linear. If the true relationship has interactions or curvature, ridge is a stable estimate of the wrong shape.

The one-line takeaway

Hoerl and Kennard proved the heretical result that an estimator which is deliberately biased can be closer to the truth than the unbiased one, by adding a penalty on large coefficients that trades a little bias for a large reduction in variance, and in doing so laid the foundation for every regularized model used in quantitative finance today.