Adam and Adaptive Optimizers

A single global learning rate is a compromise across every parameter in a network at once, and different parameters can badly disagree about what rate suits them. A parameter whose gradient is huge and jumpy needs small, cautious steps; a parameter whose gradient is small and steady could safely take bigger ones. Plain gradient descent and even plain momentum apply the same step size to both, which means the rate is always wrong for somebody. Adam fixes this by giving every parameter its own adaptive step size, updated automatically as training goes.

The analogy: a thermostat that also watches how jumpy the room is

A basic thermostat adjusts temperature by a fixed amount per correction, regardless of context. A smarter one watches not just the current reading but how erratically the temperature has been swinging lately — in a drafty room with wild swings, it makes small, cautious adjustments to avoid overcorrecting into an oscillation; in a stable room, it can afford bigger, more confident adjustments. Adam does exactly this per parameter: it tracks not only the recent average direction of the gradient (like momentum) but also the recent average size of the gradient, and divides its step by that size — large, volatile gradients get dampened, small, steady ones get amplified relative to a fixed rate.

The maths: two running averages, then a ratio

Adam keeps two exponentially weighted averages per parameter: $m_t$, tracking the mean gradient (like momentum), and $v_t$, tracking the mean squared gradient (a running estimate of how large the gradient tends to be, in either direction):

$$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$

In words: $m_t$ is "which way has this parameter's gradient generally been pointing," $v_t$ is "how big has this parameter's gradient generally been, squared so sign doesn't matter." The update then divides the momentum term by the square root of the magnitude term:

$$\theta_t = \theta_{t-1} - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}$$

(with $\hat m_t, \hat v_t$ small bias-corrected versions of $m_t, v_t$, and $\epsilon$ a tiny constant preventing division by zero). In words: a parameter whose gradients have recently been large gets its step shrunk by that same large size, and a parameter whose gradients have recently been small gets a relatively larger step — the division automatically equalises the effective step size across parameters with very different gradient scales.

Worked example 1: a noisy parameter versus a calm one

Parameter A has recent squared gradients averaging $v_A = 100$ (large, volatile gradients); parameter B has $v_B = 0.01$ (small, calm gradients). Both currently have $\hat m = 1$ and use $\eta = 0.1$, $\epsilon \approx 0$. Parameter A's step: $0.1 \times 1/\sqrt{100} = 0.1/10 = 0.01$. Parameter B's step: $0.1 \times 1/\sqrt{0.01} = 0.1/0.1 = 1.0$. Despite an identical raw momentum term, A moves by $0.01$ and B moves by $1.0$ — a 100x difference, entirely from Adam noticing that A's gradients are inherently 100x larger in magnitude and dividing that scale back out.

Worked example 2: bias correction near the start of training

At $t=1$, with $\beta_2 = 0.999$ and an initial $v_0 = 0$, a single gradient of $g_1 = 2$ gives $v_1 = 0.999(0) + 0.001(4) = 0.004$. That's a severe underestimate of the true squared-gradient scale, since $v$ started at zero and has barely moved from it after one step. Adam's bias correction divides by $(1-\beta_2^t) = 1 - 0.999^1 = 0.001$: $\hat v_1 = 0.004/0.001 = 4$ — recovering the actual value $g_1^2 = 4$ exactly. Without this correction, early steps would be wildly overconfident (dividing by a near-zero $v$ produces a huge, unstable step); the correction exists specifically to fix this cold-start problem.

Compare the paths in the explorer above: Adam's per-parameter scaling is designed to avoid both the slow crawl of too-small a learning rate and the overshoot of too-large a one, by letting each parameter effectively pick its own point on that trade-off.

Both $m_t$ and $v_t$ are exponentially weighted running averages, settling toward their true values the way the running average in the explorer above settles toward the truth — Adam's bias correction (worked example 2) exists purely to fix the slow start of exactly this kind of average.

What this means in practice

Adam is the default optimiser for the large majority of deep learning training today because it's robust to a poorly chosen global learning rate and handles the wildly different gradient scales common across a deep network's layers with no manual intervention. Its main weakness is that the adaptive per-parameter scaling can sometimes generalise slightly worse than well-tuned plain SGD with momentum, which is why some final-stage training runs switch to SGD once Adam has gotten the model into a good region.