The Geometry of L1 vs L2 Penalties

Both L1 (lasso) and L2 (ridge) regularization add a penalty term to a regression's loss to discourage large weights, and both are often taught as "roughly the same idea, different exponent." They are not the same in the one way that matters most in practice: L2 shrinks every weight a little, while L1 shrinks some weights to exactly zero, deleting features from the model entirely. Two formulas that look one exponent apart produce completely different kinds of models, and the reason is not algebra — it's geometry.

The analogy: a rubber band pulling toward a diamond vs. a circle

Picture the loss function's best-fit point sitting somewhere in weight-space, and a "budget region" around the origin the regularized solution must stay inside — a shrinking balloon squeezed onto the loss surface's contours. If that region is a smooth circle (L2), the point where the loss contours first touch it can land anywhere on the edge, including points with every coordinate nonzero. If it's a diamond with sharp corners on the coordinate axes (L1), the loss contours are disproportionately likely to first touch at a corner — and a corner on an axis means one coordinate is exactly zero. The corners are where sparsity comes from; a circle has no corners.

The formal setup

Both add a penalty to the ordinary least-squares loss, controlled by strength $\lambda$:

$$\text{Ridge (L2): } \sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_j w_j^2 \qquad \text{Lasso (L1): } \sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_j |w_j|$$

In words: both penalize how large the weights $w_j$ are, but L2 penalizes the square of each weight (a smooth, ever-steepening cost as a weight grows) while L1 penalizes the absolute value (a cost that grows at a constant rate, with a sharp kink exactly at zero). That kink is the algebraic fingerprint of the diamond's corner: the L1 penalty's derivative does not vanish smoothly as a weight approaches zero — it stays at a constant $\pm\lambda$ right up to the corner, which is enough to shove a small weight the rest of the way to exactly $0$, something the ever-softening L2 penalty near zero never has the force to do.

Worked example 1: single-weight shrinkage by hand

Take a small weight $w=0.1$ with penalty strength $\lambda=1$ and step size $\eta=0.5$. Ridge's gradient contribution is $2\lambda w = 2(1)(0.1) = 0.2$, so the update is $w \to 0.1 - 0.5(0.2) = 0.09$ — a small pull, and as $w$ keeps shrinking the pull shrinks proportionally with it, so it takes infinitely many tiny steps to actually reach zero. Lasso's gradient contribution is a constant $\lambda = 1$ regardless of $w$'s size: $w \to 0.1 - 0.5(1) = -0.4$, which a real implementation clips to exactly $0$ rather than overshooting past it. The constant-force penalty reaches zero in one step from small $w$; the proportional-force one never quite does.

Worked example 2: which features survive

A model with 5 correlated momentum features is fit with lasso at a moderate $\lambda$. Because the diamond's corners sit on individual axes, lasso tends to pick one representative per correlated cluster and zero out the rest — fitted weights might come out $(0.8, 0, 0, 0, 0)$, an automatic feature selector. Ridge on the same data instead spreads the credit: roughly $(0.2, 0.2, 0.2, 0.2, 0.2)$, keeping all five active but shrunk. Neither is "wrong" — lasso gives an interpretable, sparse model; ridge gives a stable, evenly-shared one — but they are structurally different outputs from a nearly identical formula.

Drag the exponent above between 1 (a cone, like L1's absolute-value shape) and 2 (a smooth bowl, like L2's square) and watch the corner at the origin appear and disappear — that corner is the entire mechanism behind sparsity.

What this means in practice

Use L1 when you suspect only a handful of your features actually matter and want the model to tell you which — a sparse factor model with 3 nonzero loadings out of 50 candidates is far easier to reason about and monitor than one with 50 small nonzero weights. Use L2 when features are genuinely all somewhat informative and correlated, since it avoids the somewhat arbitrary "pick one, zero the rest" behavior lasso exhibits among near-duplicate features. Elastic net, a weighted combination of both penalties, is the common practical compromise.