RMSProp and AdaGrad

Plain gradient descent uses one learning rate for every parameter in the model. But some parameters — say, a weight connected to a rare input feature — only ever get small, infrequent gradient updates, while others get hit with large gradients constantly. A single shared learning rate is a compromise that is too small for the rarely-updated parameters and possibly too large for the frequently-updated ones. Adaptive learning-rate methods give each parameter its own effective step size, shrunk or grown based on that parameter's own gradient history.

The analogy: a thermostat per room, not one for the house

A single thermostat for an entire building has to compromise: too hot in some rooms, too cold in others, because it can't see that the south-facing room gets more sun than the basement. Now imagine every room has its own thermostat that watches its own recent temperature swings and adjusts its own heater accordingly — a room that has been swinging wildly gets gentler, smaller corrections; a room that barely changes gets bigger ones. AdaGrad and RMSProp do exactly this per parameter: each one tracks its own history of gradient sizes and divides its own learning rate by a measure of how large those gradients have typically been.

AdaGrad: divide by the running total, forever

AdaGrad keeps a running sum of the squares of every past gradient for each parameter, then divides that parameter's learning rate by the square root of that sum:

$$G_t = G_{t-1} + g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}}\, g_t$$

In words: $G_t$ is the running total of squared gradients ($g_t$) seen so far for this parameter; the update subtracts the learning rate ($\eta$) divided by the square root of that running total (plus a tiny $\epsilon$ to avoid dividing by zero), times the current gradient. A parameter with a history of large gradients ends up dividing by a large number, so its effective step shrinks; a parameter with small gradient history keeps a nearly-full-size step.

The flaw: because $G_t$ only ever grows, the effective learning rate only ever shrinks, monotonically, for the whole run — eventually it can shrink so much that learning effectively stalls, even on parameters that could still use larger updates.

RMSProp: forget the distant past

RMSProp fixes that by replacing the running sum with a running exponential average, so old gradients fade out instead of accumulating forever:

$$E_t = \beta E_{t-1} + (1-\beta) g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E_t + \epsilon}}\, g_t$$

Here $E_t$ is a moving average of squared gradients, weighted by a decay factor $\beta$ (commonly $0.9$) that keeps recent gradients dominant and lets older ones decay away. This lets the effective step size grow back up if a parameter's gradients quiet down later in training, instead of being permanently shrunk by early, large gradients.

Worked example 1: AdaGrad's ever-shrinking step

A parameter sees gradients $g_1 = 4$, $g_2 = 3$, $g_3 = 2$ over three steps, with $\eta = 1$, $\epsilon \approx 0$. Running sums: $G_1 = 16$, $G_2 = 16+9=25$, $G_3 = 25+4=29$. Effective step divisors: $\sqrt{16}=4$, $\sqrt{25}=5$, $\sqrt{29}\approx5.39$. So the effective learning rate goes $1/4=0.25$, then $1/5=0.2$, then $1/5.39\approx0.186$ — steadily shrinking even though the raw gradients themselves are also shrinking, compounding the slowdown.

Worked example 2: RMSProp lets the step recover

Same gradients, RMSProp with $\beta=0.9$, $E_0=0$: $E_1 = 0.9(0)+0.1(16)=1.6$; $E_2=0.9(1.6)+0.1(9)=2.34$; $E_3=0.9(2.34)+0.1(4)=2.506$. Divisors: $\sqrt{1.6}\approx1.26$, $\sqrt{2.34}\approx1.53$, $\sqrt{2.506}\approx1.58$ — the divisor grows far more slowly than AdaGrad's, since old squared gradients are down-weighted rather than kept in full. If a later gradient were tiny, $E_t$ would shrink back down and the effective rate would rise again — something AdaGrad's ever-growing $G_t$ can never do.

What this means in practice

RMSProp (and its close relative Adam, which adds momentum on top) is the practical default for training deep networks — AdaGrad's monotonic shrinkage makes it a poor fit for long training runs, though it still works well for short, sparse-gradient problems like some linear models on sparse text features, which is what it was originally designed for.

Practice

1. A parameter has been getting tiny gradients for 1,000 steps, then suddenly gets a large one. Which method reacts faster: AdaGrad or RMSProp? Why?
2. With $\beta = 0.99$ instead of $0.9$, does RMSProp forget the past faster or slower?
3. Explain in one sentence why AdaGrad's monotonic shrinkage is actually a reasonable design for sparse features.