AdamW and Decoupled Weight Decay

Weight decay is a simple, old idea: at every update, shrink every weight slightly toward zero, which discourages the model from relying on any single weight too heavily and tends to improve generalization. For plain gradient descent, adding weight decay is equivalent to adding an $L_2$ penalty to the loss — they amount to the same update. Adam, the most widely used optimizer in deep learning, also lets you add weight decay this way. The surprising finding is that for Adam, doing it the "add to the loss" way is subtly wrong, and fixing it — decoupling weight decay from the gradient-based update entirely — is what AdamW does.

The analogy: rent that scales with your income versus rent that doesn't

Imagine a landlord who charges rent proportional to how volatile your recent income has been — a bigger discount in a slow month, a bigger charge in a busy one. That is what happens if you fold weight decay into the loss and let Adam's adaptive step-size machinery process it: Adam divides every gradient, including the "shrink toward zero" signal from the decay term, by each parameter's own adaptive scale. So a parameter that has been getting large, noisy gradients has its decay diluted, while a parameter with small gradients gets decayed relatively harder — a coupling nobody asked for. AdamW instead charges flat rent, in a fixed amount, decoupled from how your income has been behaving: the decay is applied as its own separate, plain shrinkage step, untouched by Adam's adaptive scaling.

The mechanics

Ordinary "$L_2$-regularized Adam" adds the decay to the gradient before Adam's adaptive scaling touches it:

$$g_t \leftarrow g_t + \lambda \theta_t, \qquad \theta_{t+1} = \theta_t - \eta \cdot \frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}$$

Here $\lambda$ is the decay strength, $\theta_t$ the current weight, and $\hat m_t, \hat v_t$ are Adam's momentum and adaptive-variance estimates — the decay term $\lambda\theta_t$ gets folded into $g_t$ and then divided by $\sqrt{\hat v_t}$ along with everything else, so its effective strength varies parameter by parameter.

AdamW instead keeps the decay completely separate from that division:

$$\theta_{t+1} = \theta_t - \eta \cdot \frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} - \eta \lambda \theta_t$$

In words: take Adam's usual adaptively-scaled step, and then, as a fully separate operation, shrink the weight by a fixed fraction $\eta\lambda$ — the same fractional shrinkage for every parameter, regardless of that parameter's gradient history.

Worked example 1: the coupling effect

Two weights, both currently at value $\theta = 2.0$, with decay $\lambda = 0.01$ and learning rate $\eta = 0.1$. Weight A has a large adaptive scale $\sqrt{\hat v_t} = 10$ (noisy gradients); weight B has a small one, $\sqrt{\hat v_t} = 1$ (quiet gradients). Under $L_2$-in-the-loss Adam, the decay contribution to the gradient is $\lambda\theta = 0.01 \times 2.0 = 0.02$ for both, but it then gets divided: A's effective decay push is $0.02/10 = 0.002$, B's is $0.02/1 = 0.02$ — a tenfold difference in how hard the two weights are actually decayed, purely because of unrelated gradient noise.

Worked example 2: AdamW's flat decay

Same two weights under AdamW: the decay step is $\eta\lambda\theta = 0.1 \times 0.01 \times 2.0 = 0.002$ for both A and B, regardless of their adaptive scales — every weight gets shrunk by the same $0.2\%$ of its value this step. That predictable, uniform shrinkage is what makes weight decay behave the way it is supposed to: a knob for overall model complexity, not one whose effect depends unpredictably on each parameter's recent gradient noise.

What this means in practice

AdamW is now the default optimizer for training transformers and most modern deep networks precisely because of this fix — Loshchilov and Hutter showed it closes a measurable generalization gap that plain "Adam with $L_2$ regularization" left on the table, at no extra computational cost. If a training recipe specifies "Adam" with a weight decay hyperparameter, check whether the actual implementation is coupled or decoupled — most modern deep learning frameworks now default to AdamW under the hood, but older or custom code may not.

Practice

1. Two weights have adaptive scales $\sqrt{\hat v_t}=2$ and $\sqrt{\hat v_t}=20$. Under $L_2$-in-loss Adam with $\lambda\theta=0.04$, what is each weight's effective decay push?
2. Under AdamW with $\eta=0.05$, $\lambda=0.02$, $\theta=3.0$, what is the flat decay applied?
3. Why does plain SGD not have this coupling problem in the first place?