SCAD and MCP Non-Convex Penalties

Think of the lasso penalty as a landlord who charges every tenant the same rent increase regardless of income: it shrinks every coefficient toward zero by a fixed amount, which is great for pushing genuinely useless features out of a model but unfair to features with a real, large effect — those get dragged down too, just for being large. SCAD (smoothly clipped absolute deviation) and MCP (minimax concave penalty) are built to behave like the lasso for small coefficients, where shrinkage is doing useful work killing noise, and then taper off so that once a coefficient is clearly large, it stops being shrunk at all.

Both achieve this by using a penalty whose slope decreases as the coefficient grows, rather than the lasso's constant slope. Small coefficients still get pulled toward zero the way the lasso would; past a threshold, the penalty flattens out, so the optimizer no longer has any incentive to shrink further. The result is a model that keeps the lasso's sparsity — many coefficients set exactly to zero — while leaving the surviving, clearly-important coefficients close to their true, unshrunk size instead of systematically underestimating them.

Concretely, if the true coefficient is 5 and the lasso with a tuning penalty of 0.5 would shrink it to roughly 4.5, a SCAD penalty tuned so its flattening point sits below 5 leaves that coefficient essentially untouched, while a smaller true coefficient of 0.3 still gets shrunk toward zero exactly as the lasso would.

The cost is that the optimization problem is no longer convex, so fitting one takes more care than a lasso fit (coordinate descent still works but needs a good starting point) and there's no single global minimum guaranteed.