Regularization as a Prior

Fit a regression with 40 candidate features and 60 rows of data and least squares will hand you a beautiful answer: coefficients that reproduce your training set almost perfectly and that mean nothing. Refit next month on new data and every coefficient moves, several flip sign, and the ones that looked strongest are the ones that moved most.

The data alone cannot pin the answer down, so something else has to. In practice everyone reaches for the same fix — add a penalty term that punishes big coefficients and rerun. It works, and it is usually taught as a trick: large coefficients are suspicious, so tax them. That framing leaves the important questions unanswerable. Why square the coefficients rather than cube them? Why does the absolute-value version set things to exactly zero when the squared version never does? How big should the penalty be?

All three have crisp answers once you see what the penalty actually is: a statement of what you believed about the coefficients before the data arrived.

The analogy: a new trader's first month

A trader joins your desk and makes 4% in his first month. Annualised, that is roughly 60%. Do you write that into next year's revenue plan?

Obviously not. You blend it with what you knew beforehand: most traders on this desk make something close to zero, and one month is a tiny sample. So you shade the estimate down hard — perhaps to 5% or 8% annualised. If he keeps it up for two years you shade much less, because now the evidence is loud enough to overrule your prior expectation.

Notice you did three things. You started from a belief centred on zero. You measured something. You produced an answer between them, closer to the data as the data got stronger and closer to zero as your prior belief got firmer. That is the entire content of ridge regression, and the "penalty" is just the algebraic residue of the prior.

From prior to penalty

Bayes' rule says the belief you should hold after seeing data $D$ is proportional to how well each coefficient vector $w$ explains that data, times how plausible you thought $w$ was beforehand:

$$p(w \mid D) \;\propto\; p(D \mid w)\,p(w)$$

In words: posterior belief = evidence × prior belief. The symbol $p(D \mid w)$ is the likelihood — the probability of the data you actually saw, if $w$ were the truth. The symbol $p(w)$ is the prior — how plausible $w$ looked before any data.

Maximising a product is awkward, so take negative logarithms, which turns products into sums and turns "find the biggest" into "find the smallest":

$$\hat{w} = \arg\min_{w} \; \Big[\underbrace{-\log p(D \mid w)}_{\text{a loss function}} \;+\; \underbrace{-\log p(w)}_{\text{a penalty}}\Big]$$

That is the whole identity. The loss you minimise is the negative log-likelihood, and the penalty you add is the negative log of your prior. Every penalty in machine learning is some prior read backwards.

Two priors dominate. A Gaussian prior $w \sim N(0, \tau^2)$ has $-\log p(w) = w^2/(2\tau^2)$ plus a constant — a squared penalty, so ridge. A Laplace prior with scale $b$, whose density falls off like $e^{-|w|/b}$, has $-\log p(w) = |w|/b$ — an absolute-value penalty, so lasso. The penalty strength is not arbitrary either: with Gaussian noise of variance $\sigma^2$ in the data, the ridge parameter is $\lambda = \sigma^2/\tau^2$, the ratio of how noisy your measurements are to how uncertain you were beforehand.

Worked example 1: shrinkage, computed

Estimate one number $w$ from $n = 10$ noisy readings whose average came out at $\bar{y} = 4.0$. The noise variance is $\sigma^2 = 4$. Your prior is $w \sim N(0, \tau^2)$ with $\tau^2 = 1$ — you genuinely expected something within about $\pm 1$ of zero.

Step 1 — the penalty strength. $\lambda = \sigma^2/\tau^2 = 4/1 = 4$.

Step 2 — minimise. The objective is $\sum_i (y_i - w)^2 + \lambda w^2$. Differentiate and set to zero: $-2\sum_i (y_i - w) + 2\lambda w = 0$, so $n(\bar{y} - w) = \lambda w$, giving

$$\hat{w} = \frac{n\bar{y}}{n + \lambda} = \frac{10 \times 4.0}{10 + 4} = \frac{40}{14} = 2.857$$

Step 3 — check it against Bayes directly. Posterior precision (the inverse of variance) adds: $n/\sigma^2 + 1/\tau^2 = 2.5 + 1 = 3.5$. Posterior mean $= (n/\sigma^2)\bar{y} / 3.5 = 2.5 \times 4.0 / 3.5 = 2.857$. Identical. The penalised estimate is the peak of the posterior — nothing was approximated.

Step 4 — see the dials move. The shrinkage factor is $n/(n+\lambda) = 10/14 = 0.714$; the estimate keeps 71.4% of what the data said. With $n = 100$ readings instead, $\hat{w} = 400/104 = 3.846$ and the factor rises to 0.962 — loud evidence, prior barely heard. Tighten the prior instead to $\tau^2 = 0.25$, so $\lambda = 16$, and $\hat{w} = 40/26 = 1.538$. Same data, firmer belief, harder shrink.

This is the trader story with numbers attached: 4.0 was the measurement, 0 was the prior, 2.857 is where the argument settled.

The explorer below is the same mechanism with a coin instead of a coefficient. Drag the prior to be tight or diffuse, then add observations, and watch the posterior slide from the prior towards the data as evidence accumulates.

Worked example 2: why lasso hits exactly zero and ridge does not

Take the simplest possible case — one coefficient, unit noise variance, and $z$ standing for what plain least squares would have chosen.

Ridge. Minimise $\tfrac{1}{2}(z - w)^2 + \tfrac{\lambda}{2}w^2$. The derivative is $(w - z) + \lambda w$, zero at

$$\hat{w}_{\text{ridge}} = \frac{z}{1+\lambda}$$

With $\lambda = 0.20$: an OLS estimate of $z = 0.30$ becomes $0.30/1.2 = 0.250$, and $z = 0.12$ becomes $0.100$. Everything is scaled by the same factor and nothing reaches zero.

Lasso. Minimise $\tfrac{1}{2}(z-w)^2 + \lambda|w|$. For $w > 0$ the derivative is $(w - z) + \lambda$, zero at $w = z - \lambda$; if that is negative the minimum sits at the corner instead. So

$$\hat{w}_{\text{lasso}} = \text{sign}(z)\,\max(|z| - \lambda,\; 0)$$

With $\lambda = 0.20$: $z = 0.30$ becomes $0.30 - 0.20 = 0.100$, and $z = 0.12$ becomes exactly $0$, because $0.12 < 0.20$.

Why the difference, in one sentence. The pull a penalty exerts is its slope. The Gaussian penalty $w^2/(2\tau^2)$ has slope $w/\tau^2$, which goes to zero as $w$ approaches zero — the prior stops pushing just when it would need to push hardest. The Laplace penalty $|w|/b$ has slope $1/b$ right up to the origin — a constant force that never lets go, so any coefficient whose evidence is weaker than that force gets pinned at zero. The corner in the picture above is that constant force.

What this means in practice

You can reason about $\lambda$ instead of grid-searching it blindly. "How large did I honestly expect these coefficients to be?" gives you $\tau$, and the residual scale gives you $\sigma$. That will not beat cross-validation, but it tells you the right order of magnitude and catches the case where the best cross-validated $\lambda$ implies a prior nobody would defend.

Standardise your features first, or your prior is nonsense. One penalty applied to coefficients measured in basis points and in dollars asserts that both should be about the same size. They should not.

Do not penalise the intercept. A prior centred on zero is a claim that the effect is probably absent. You rarely believe that about the overall mean.

Choosing a penalty is choosing a belief, so choose deliberately. Expect a handful of real drivers among many candidates? That is a spike near zero with fat tails, and lasso approximates it. Expect many small correlated effects, as with most return factors? That is Gaussian, so ridge. Want both, use elastic net.

Practise it

1. With $n = 25$, $\sigma^2 = 9$, $\tau^2 = 1$ and $\bar{y} = 2.0$, compute $\lambda$ and $\hat{w}$. (Answer: $\lambda = 9$, $\hat{w} = 50/34 = 1.471$.)
2. How large must $n$ be before the shrinkage factor $n/(n+\lambda)$ exceeds 0.95 when $\lambda = 4$? (Answer: $n > 76$.)
3. Under lasso with $\lambda = 0.15$, which of the OLS estimates $-0.42$, $0.09$, $0.15$, $-0.31$ survive, and at what values? (Answer: $-0.27$, $0$, $0$, $-0.16$.)
4. A colleague sets $\lambda = 100$ on standardised features and gets good cross-validated error. What prior standard deviation $\tau$ does that imply if the residual variance is 4, and would you defend that belief out loud?