The Adaptive Lasso

The ordinary lasso shrinks every regression coefficient toward zero by the same fixed amount. That's what gives it automatic feature selection — some coefficients hit exactly zero — but it has a side effect: a coefficient that's genuinely large gets shrunk almost as hard as one that should be zero, biasing real effects toward zero even after the lasso has correctly kept them.

The analogy: a flat tax versus a means-tested one

A flat tax takes the same percentage from everyone, so a large, legitimate income is taxed almost as hard, proportionally, as income that looks like noise. A means-tested system instead sets a different rate per person based on a prior read of their situation: light taxation for incomes that look large and real, heavy taxation for incomes that look like they shouldn't be there. The adaptive lasso does this to coefficients, using a first-pass fit to decide how hard to "tax" each one.

The mechanics: weights built from a first-pass fit

The ordinary lasso penalizes every coefficient equally: $\min_\beta \sum_i(y_i - x_i^\top\beta)^2 + \lambda\sum_j|\beta_j|$. The adaptive lasso instead uses per-coefficient weights $w_j$ built from an initial estimate $\hat\beta_j^{(0)}$ (typically OLS or ridge):

$$w_j = \frac{1}{|\hat\beta_j^{(0)}|^{\gamma}}, \qquad \min_\beta \ \sum_i (y_i - x_i^\top \beta)^2 + \lambda \sum_j w_j |\beta_j| .$$

In plain English: a coefficient that looked large in the first pass gets a small weight, so it's barely penalized and survives close to its true size; a coefficient that looked small or noisy gets a large weight and is pushed to zero.

Worked example: one real signal, one noise signal

An initial OLS fit on five candidate signals gives $\hat\beta^{(0)} = (0.40, 0.02, -0.35, 0.01, 0.03)$. With $\gamma=1$, the weights are $(2.5, 50, 2.86, 100, 33.3)$. Signals A and C face light penalty multipliers and survive the second-stage fit close to their original size; B, D and E face multipliers of 33–100, and even a modest $\lambda$ pushes them to exactly zero. An ordinary lasso, penalizing all five equally, would have shrunk A and C by the same proportion used on the noise signals.

Drag the exponent above and notice how a steep power curve maps a small ratio of "weight" to a huge penalty multiplier — that steepness is exactly what turns a modest first-pass coefficient gap into a very different second-stage penalty.

What this means in practice

The adaptive lasso matters most when you care about the size of surviving coefficients, not just which variables get selected — estimating how strongly a factor predicts returns, rather than merely screening a long candidate list. It needs a reasonable first-pass estimate to build the weights from, so it needs more data than the plain lasso to work well.