Ridge and LASSO Regularization

Regularization is the deliberate introduction of bias to reduce variance, the bias–variance tradeoff turned into a tool. OLS is unbiased and, by Gauss–Markov, best among unbiased linear estimators; but "best unbiased" is a low bar when the unbiased variance is enormous, which is exactly the situation in finance: many correlated predictors, short samples, and tiny signal-to-noise. Ridge and LASSO give up unbiasedness to buy a large drop in variance, and they are the default estimators whenever the predictor count is large relative to the information in the data.

The two penalties

Both add a penalty on coefficient size to the least-squares objective. Ridge penalizes the squared ($L_2$) norm:

$\hat\beta_{\text{ridge}} = \arg\min_\beta\ \lVert y - X\beta\rVert_2^2 + \lambda\lVert\beta\rVert_2^2, \qquad \lVert\beta\rVert_2^2 = \sum_j \beta_j^2.$

LASSO penalizes the absolute ($L_1$) norm:

$\hat\beta_{\text{lasso}} = \arg\min_\beta\ \lVert y - X\beta\rVert_2^2 + \lambda\lVert\beta\rVert_1, \qquad \lVert\beta\rVert_1 = \sum_j |\beta_j|.$

The tuning parameter $\lambda \ge 0$ controls the strength: $\lambda = 0$ recovers OLS; $\lambda \to \infty$ shrinks all coefficients to zero. You choose $\lambda$ by cross-validation, and predictors must be standardized first so the penalty treats them comparably.

Ridge has a closed form, and regularizes the singularity

Ridge's objective is smooth, so setting the gradient to zero gives

$\boxed{\hat\beta_{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top y.}$

Compare to OLS's $(X^\top X)^{-1}X^\top y$: ridge adds $\lambda I$ to $X^\top X$ before inverting. This is the direct antidote to Multicollinearity, recall that collinearity makes $X^\top X$ near-singular with tiny eigenvalues that blow up the inverse. Adding $\lambda I$ lifts every eigenvalue by $\lambda$, so $\lambda_{\min} + \lambda$ replaces $\lambda_{\min}$ in the denominator, taming the variance. In the eigenbasis, ridge multiplies the OLS coefficient along direction $j$ by the shrinkage factor $\lambda_j/(\lambda_j + \lambda)$, barely touching high-variance (well-identified) directions while heavily shrinking the low-variance (collinear, poorly-identified) ones. It shrinks smoothly toward zero but never sets a coefficient exactly to zero.

LASSO selects, the geometry of the corner

LASSO's crucial difference is that it drives some coefficients to exactly zero, performing variable selection and yielding a sparse model. The reason is geometric. Both methods can be written as constrained problems, minimize the residual sum of squares subject to $\lVert\beta\rVert_2^2 \le t$ (ridge) or $\lVert\beta\rVert_1 \le t$ (LASSO). The elliptical RSS contours expand until they touch the constraint region. The $L_2$ ball is round, so the tangency generically happens at a point with all coordinates nonzero, ridge shrinks but keeps everything. The $L_1$ ball is a diamond with sharp corners on the axes, and the expanding ellipse very often first touches a corner, where one or more coefficients are exactly zero. Higher-dimensional $L_1$ balls have corners, edges, and faces on the coordinate subspaces, making sparse solutions the rule, not the exception. That corner geometry is the whole story of why $L_1$ selects and $L_2$ does not.

A useful special case: with an orthonormal design, LASSO is soft-thresholding, $\hat\beta_j^{\text{lasso}} = \operatorname{sign}(\hat\beta_j^{\text{OLS}})(|\hat\beta_j^{\text{OLS}}| - \lambda/2)_+$, shift each coefficient toward zero by $\lambda/2$ and clip at zero, while ridge is proportional shrinkage $\hat\beta_j^{\text{OLS}}/(1+\lambda)$. Elastic net, $\lambda_1\lVert\beta\rVert_1 + \lambda_2\lVert\beta\rVert_2^2$, blends the two: it selects like LASSO but handles groups of correlated predictors more gracefully (LASSO tends to pick one of a collinear group arbitrarily).

Why finance needs it: the bias–variance payoff

The bias–variance decomposition is the justification. Ridge/LASSO are biased ($\mathbb{E}[\hat\beta]\ne\beta$), but there always exists a $\lambda > 0$ for which the increase in squared bias is more than offset by the decrease in variance, lowering MSE, this is a theorem for ridge, not a hope. In finance the conditions that make the trade favorable are everywhere: predictors (factors, technicals, alternative-data features) are numerous and highly collinear, samples are short, and the true signal is faint. Unregularized OLS in that setting overfits, high in-sample $R^2$, negative out-of-sample performance (see Overfitting). Shrinkage stabilizes coefficients, prevents sign-flipping, and improves out-of-sample forecasts, which is why penalized regression, not OLS, is the practical default for signal construction. It is the same instinct as covariance Shrinkage in portfolio construction.

Worked example: 50 collinear factors, 200 months

You have 50 candidate return predictors, many correlated, and 200 monthly observations. OLS gives wild coefficients, some huge, several with implausible signs, and an in-sample $R^2$ of 0.35 that collapses to negative out-of-sample. Ridge with a cross-validated $\lambda$ pulls the coefficients toward zero, kills the sign instability, and yields a small but positive out-of-sample $R^2$. LASSO at its CV-optimal $\lambda$ zeroes 40 of the 50, leaving a sparse, interpretable 10-factor model you can actually reason about and trade with lower turnover. Same data, the regularized models generalize, the OLS model does not.

Failure modes in financial data

• LASSO instability under collinearity. When predictors are highly correlated, LASSO's choice of which one to keep is unstable across samples, a small data change swaps the selected factor. Elastic net or ridge is steadier when you care about the coefficients themselves.
• Cross-validation leakage in time series. Standard $k$-fold CV shuffles time and leaks future into past, badly under-selecting $\lambda$; use forward-chaining / Walk-Forward Analysis CV that respects the time order.
• Standardization and stationarity. The penalty is scale-sensitive, so features must be standardized on training data only; and if the underlying relationships drift, the CV-chosen $\lambda$ and selected set are themselves non-stationary.
• Naive inference. Post-selection $t$-statistics on the LASSO-chosen coefficients are invalid (the selection used the data); honest inference needs post-selection or debiased-LASSO methods.

In interviews

Write both objectives, ridge with $\lambda\lVert\beta\rVert_2^2$, LASSO with $\lambda\lVert\beta\rVert_1$, and give ridge's closed form $(X^\top X + \lambda I)^{-1}X^\top y$, noting that the $\lambda I$ is exactly what cures a near-singular $X^\top X$. The signature question is "why does LASSO produce sparse solutions and ridge not?", answer with the constraint geometry: the $L_1$ diamond has corners on the axes that the RSS ellipse tends to hit, zeroing coefficients, while the round $L_2$ ball has no corners. Ground the whole thing in bias–variance: you accept bias to cut variance and lower MSE, which is worthwhile precisely in finance's many-collinear-predictors, low-signal, short-sample regime. A strong closer connects ridge to a Gaussian prior and LASSO to a Laplace prior, regularization is Bayesian MAP estimation. See Bayesian Inference.