The Regularization Path

Imagine turning a dial from "penalize every coefficient into oblivion" down to "no penalty at all" and, at each setting, refitting a regression. At the strong end every coefficient sits at zero; as you loosen the dial, variables switch on one by one, each growing from zero once the penalty is weak enough to let it matter. Plotting every coefficient's value against the dial setting produces the regularization path — a single picture that shows you not just which variables end up in the model, but the order in which the data insists they matter.

This only bites for lasso-family penalties (an $L_1$ penalty on the coefficients), because that penalty has a kink at zero that forces coefficients to hit exactly zero and stay there until the dial loosens enough. A ridge ($L_2$) penalty shrinks everything continuously toward zero without ever fully zeroing anything out, so its "path" is just a smooth shrinkage — informative, but it never tells you which variables to drop.

Worked example. Fitting a return-prediction model with five candidate factors (value, momentum, size, quality, low-vol) across a lasso path: at the strongest penalty all five coefficients are 0. Loosening the dial, momentum switches on first, then value, then size — quality and low-vol never switch on even at the weakest penalty tried. That ordering is itself information: momentum and value are the factors the data leans on hardest, and quality/low-vol are redundant given the others already in the model, not necessarily useless on their own.

The path is normally read alongside cross-validated error to pick one penalty strength, but the full path is worth inspecting even after you've chosen a model — a variable that only enters at the very weakest penalty setting is a fragile one to trust.