The Adaptive Lasso

The ordinary lasso shrinks every regression coefficient toward zero by the same fixed amount. That's what gives it automatic feature selection, some coefficients hit exactly zero, but it has a side effect: a coefficient that's genuinely large gets shrunk almost as hard as one that should be zero, biasing real effects toward zero even after the lasso has correctly kept them.

The analogy: a flat tax versus a means-tested one

A flat tax takes the same percentage from everyone, so a large, legitimate income is taxed almost as hard, proportionally, as income that looks like noise. A means-tested system instead sets a different rate per person based on a prior read of their situation: light taxation for incomes that look large and real, heavy taxation for incomes that look like they shouldn't be there. The adaptive lasso does this to coefficients, using a first-pass fit to decide how hard to "tax" each one.

The mechanics: weights built from a first-pass fit

The ordinary lasso penalizes every coefficient equally: $\min_\beta \sum_i(y_i - x_i^\top\beta)^2 + \lambda\sum_j|\beta_j|$. The adaptive lasso instead uses per-coefficient weights $w_j$ built from an initial estimate $\hat\beta_j^{(0)}$ (typically OLS or ridge):

In plain English: a coefficient that looked large in the first pass gets a small weight, so it's barely penalized and survives close to its true size; a coefficient that looked small or noisy gets a large weight and is pushed to zero.

Worked example: one real signal, one noise signal

An initial OLS fit on five candidate signals gives $\hat\beta^{(0)} = (0.40, 0.02, -0.35, 0.01, 0.03)$. With $\gamma=1$, the weights are $(2.5, 50, 2.86, 100, 33.3)$. Signals A and C face light penalty multipliers and survive the second-stage fit close to their original size; B, D and E face multipliers of 33–100, and even a modest $\lambda$ pushes them to exactly zero. An ordinary lasso, penalizing all five equally, would have shrunk A and C by the same proportion used on the noise signals.

Drag the exponent above and notice how a steep power curve maps a small ratio of "weight" to a huge penalty multiplier, that steepness is exactly what turns a modest first-pass coefficient gap into a very different second-stage penalty.

What this means in practice

The adaptive lasso matters most when you care about the size of surviving coefficients, not just which variables get selected, estimating how strongly a factor predicts returns, rather than merely screening a long candidate list. It needs a reasonable first-pass estimate to build the weights from, so it needs more data than the plain lasso to work well.