Weight Decay vs L2 Regularization

L2 regularization adds a penalty $\frac{\lambda}{2}\|w\|^2$ to the loss function, so the optimizer is minimizing loss plus a term that punishes large weights. Weight decay is a direct update rule: at every step, shrink each weight by a fixed fraction, $w \leftarrow w - \eta \lambda w$, before applying the gradient step. Under plain stochastic gradient descent these turn out to be exactly equivalent, because the gradient of the L2 penalty term $\lambda w$ added to the loss gradient produces precisely that same shrinkage.

The equivalence breaks under Adam and similar adaptive optimizers, which rescale each parameter's update by a running estimate of that parameter's gradient magnitude:

$$w \leftarrow w - \eta \, \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}.$$

If you implement "regularization" as an L2 penalty added to the loss (as most deep learning frameworks did by default for years), the penalty's gradient $\lambda w$ gets divided by $\sqrt{\hat v}$ along with everything else — so parameters with large historical gradients get less shrinkage than intended, and parameters with small gradients get more, an unintended and parameter-dependent side effect nobody asked for. Applying weight decay as a direct multiplicative shrink instead ($w \leftarrow w - \eta\lambda w$, applied separately from the adaptive gradient step) avoids this entirely — that's the whole idea behind AdamW.

Worked example. Suppose $\lambda = 0.01$, $\eta = 0.001$, and a parameter's adaptive denominator $\sqrt{\hat v}+\epsilon = 0.1$. L2-as-penalty shrinks the weight by $\eta \cdot \lambda w / 0.1 = 0.0001w$ (10x amplified by the small denominator). Decoupled weight decay shrinks it by $\eta\lambda w = 0.00001w$ regardless of $\hat v$ — the intended, denominator-independent amount.