The Lasso: A Regression That Fires Its Own Useless Variables

You have two hundred candidate predictors of next month's stock returns. You have five thousand observations. You run a regression.

The regression will give you two hundred coefficients, and every single one of them will be non-zero. Not because all two hundred predictors matter, but because ordinary least squares is a machine that is physically incapable of saying "this variable is useless." It will always find some small non-zero weight that improves the fit a tiny bit, and that tiny bit is almost always noise.

This is a real, practical, expensive problem, and in 1996 Robert Tibshirani solved it with a change so small it fits in a single line.

The problem: too many variables, and a model that will not say no

When you have a lot of predictors relative to your data, ordinary regression falls apart in two ways.

It overfits. With enough variables, the regression can bend itself to fit the noise in your training sample perfectly, and it will then be useless out of sample. In the extreme, if you have as many predictors as observations, you can fit the data exactly, with zero error, and have learned absolutely nothing.

It becomes unstable when predictors are correlated. In finance, they always are. Half a dozen ways of measuring "value" are all telling you nearly the same thing. When two predictors are nearly identical, ordinary regression cannot tell which one deserves the credit, and it copes by assigning wild, arbitrary coefficients, often a huge positive one to one and a huge negative one to the other, that cancel out. The predictions might be alright. The coefficients are gibberish, and they will flip signs entirely if you add a few more months of data.

Meanwhile, you actually want to know which variables matter. You want a short list. Regression will not give you one.

The traditional fix was stepwise selection: add variables one at a time, keeping the ones that help, or start with all of them and drop the worst. This is a greedy search, it is unstable, and it makes a mess of your statistical inference because the selection process is itself a form of data snooping.

The key idea via analogy: a budget for coefficients

Tibshirani's idea is to change what the regression is trying to do.

Ordinary regression has one goal: make the errors as small as possible. Nothing restrains it.

The lasso adds a second goal: make the errors as small as possible, while keeping the total size of your coefficients under a budget. Specifically, add up the absolute values of all your coefficients, and require that sum to stay below some limit.

That is the whole thing. One constraint.

The analogy: imagine you are furnishing a room, and each piece of furniture is a predictor. Ordinary regression lets you buy everything, so it does, cramming the room full of junk on the grounds that each item makes the room marginally more useful. The lasso gives you a fixed budget and tells you to spend it well. Suddenly you have to make choices. The couch is clearly worth it. The decorative ceramic frog is not. You leave the frog in the shop.

Now here is the part that made this a landmark rather than a footnote. There is an older method, ridge regression, that does something very similar: it also constrains the coefficients, but it budgets the sum of the squared coefficients rather than the absolute values. Ridge shrinks everything toward zero, which helps enormously with overfitting and with correlated predictors.

But ridge never sets anything to exactly zero. It gives your useless predictors tiny coefficients, and tiny is not zero. You still have two hundred variables in your model. You still have no short list.

The lasso sets coefficients to exactly zero. Not small. Zero. The variable is gone, out of the model entirely.

Why? The reason is geometric and genuinely beautiful. The constraint "sum of absolute values stays under a budget" carves out a region with sharp corners, and those corners sit exactly on the axes, which is to say at the points where one or more coefficients are exactly zero. The optimisation process, sliding toward the best fit it can achieve within the budget, tends to bump into a corner. The squared-value constraint of ridge regression carves out a smooth round region with no corners anywhere, so there is nothing to catch on, and the solution almost never lands precisely on an axis.

Corners create zeros. That single geometric fact is why the lasso can do variable selection and ridge cannot.

Why it mattered

• It fused two separate tasks into one. Before the lasso, "shrink the coefficients to avoid overfitting" and "select which variables to keep" were different problems, handled by different, incompatible methods. Tibshirani showed that one constraint does both, simultaneously, in a single convex optimisation with a unique solution. That is an unusually elegant piece of work.
• It made high-dimensional regression possible. The lasso works even when you have far more predictors than observations, a situation in which ordinary regression does not merely perform poorly, it has no unique answer at all. That capability unlocked genomics, text analysis, and modern quantitative finance.
• It is the workhorse of signal selection in quant research. You have three hundred candidate features and a suspicion that a dozen of them matter. The lasso is the standard first tool for finding out which dozen. Gu, Kelly and Xiu use penalized linear models as one of the main contestants in their machine learning horse race for exactly this reason.
• It launched an enormous literature. The elastic net, the group lasso, the adaptive lasso, the fused lasso, and compressed sensing all descend from this paper. It is one of the most cited papers in modern statistics.

The honest limitations

• It handles correlated predictors badly, and finance is nothing but correlated predictors. If two features are nearly identical, the lasso will typically pick one of them, essentially arbitrarily, and zero out the other. Which one it picks can flip if you add a little more data. This makes the resulting variable list look decisive and authoritative when it is actually fragile, and that is a dangerous combination. Zou and Hastie invented the elastic net specifically to fix this.
• The coefficients it gives you are biased, on purpose. Shrinkage is the whole mechanism, so the coefficients on the surviving variables are systematically pulled toward zero, meaning they understate the true effect sizes. If you want to interpret the magnitude of an effect rather than just predict, this matters.
• Statistical inference after the lasso is treacherous. You cannot select variables with the lasso and then compute standard p-values on the survivors as if you had chosen them in advance. That is textbook data snooping, and the resulting p-values are meaningless. Valid post-selection inference is a hard and active research area.
• The tuning parameter has to be chosen, and it changes everything. How tight is the budget? Too tight and you throw away real signal. Too loose and you are back to overfitting. It is normally chosen by cross-validation, which in financial time series must be done with proper respect for the arrow of time, or you have simply moved the overfitting somewhere less visible.
• It is still a linear model. The lasso selects variables and shrinks coefficients, but it assumes the relationship is a straight line. If the truth is a non-linear interaction, the lasso cannot see it, no matter how well you tune it.

The one-line takeaway

Tibshirani showed that by budgeting the sum of the absolute values of a regression's coefficients rather than their squares, you create a constraint with sharp corners that pins useless variables to exactly zero, giving you overfitting control and automatic variable selection from a single line of maths.