Momentum and Nesterov Acceleration

Plain gradient descent takes a step exactly opposite the current gradient, forgets everything about where it just came from, and recomputes from scratch every time. On a loss surface shaped like a long narrow valley — steep on the sides, gently sloped along the bottom — this makes it zig-zag violently back and forth across the valley walls while creeping only slowly along the direction that actually matters. Momentum fixes this by giving the optimiser a memory.

The analogy: a ball rolling downhill, not a hiker checking a compass at every step

A hiker using plain gradient descent stops at every single step, checks a compass for the steepest downhill direction right here, and takes one step that way — nothing carries over from the last step. If the terrain has small ridges running across the true downhill direction, the hiker zig-zags across each ridge instead of making steady progress. A heavy ball rolling down the same hill behaves completely differently: it has momentum from its recent motion, so small ridges barely deflect it, while it still gradually curves toward the true downhill direction. That inertia is what momentum adds to gradient descent — not a smarter compass, but a refusal to fully forget the direction you were already moving.

The maths: an exponentially weighted running average of gradients

Momentum keeps a running "velocity" $v$ that blends the current gradient with the velocity from the previous step:

$$v_t = \beta v_{t-1} + (1-\beta)\,g_t, \qquad \theta_t = \theta_{t-1} - \eta v_t$$

Here $g_t$ is the current gradient, $\beta$ (typically around $0.9$) controls how much of the past velocity survives each step, $\eta$ is the learning rate, and $\theta$ is the parameter being updated. In words: instead of stepping directly opposite today's gradient, step opposite a blend of today's gradient and yesterday's direction of travel — consistent gradients across steps reinforce each other and the step size effectively grows, while gradients that flip back and forth (the zig-zag) partially cancel in the average and shrink. Nesterov acceleration makes one refinement: it computes the gradient not at the current position but at the position the momentum is already about to carry you to, giving the optimiser a chance to correct course slightly before fully committing to the momentum's direction.

Worked example 1: momentum cancels a zig-zag

Suppose the gradient alternates between $g = (+3, 1)$ and $g = (-3, 1)$ on successive steps — a hard left-right zig-zag in the first coordinate, consistent drift in the second. With $\beta = 0.9$, starting from $v_0=(0,0)$: after step 1, $v_1 = 0.9(0,0) + 0.1(3,1) = (0.3, 0.1)$. After step 2 with $g=(-3,1)$: $v_2 = 0.9(0.3,0.1) + 0.1(-3,1) = (-0.03, 0.19)$. After step 3, back to $g=(3,1)$: $v_3 = 0.9(-0.03,0.19)+0.1(3,1) = (0.273, 0.271)$. Notice the first coordinate is staying small and oscillating near zero (the zig-zag is being cancelled by the averaging) while the second coordinate is climbing steadily toward the consistent drift — exactly the intended behaviour.

Worked example 2: momentum accelerates on a consistent slope

Now suppose the gradient is a constant $g = (0, 2)$ every step (a steady, unchanging downhill slope), $\beta=0.9$. $v_1 = 0.1(2) = 0.2$. $v_2 = 0.9(0.2)+0.1(2) = 0.38$. $v_3 = 0.9(0.38)+0.1(2) = 0.542$. $v_4 = 0.9(0.542)+0.1(2)=0.688$. The velocity is climbing toward its steady-state value of $2$ (since at equilibrium $v=\beta v + (1-\beta)g$ gives $v=g$), meaning the effective step size on a consistent slope grows roughly toward $1/(1-\beta) = 10\times$ a single plain-gradient-descent step — this is the "rolling ball speeding up on a steady incline" effect made numeric.

Try the explorer above and compare a plain gradient-descent path against a momentum path on the same loss surface — plain descent zig-zags across a narrow valley while momentum smooths the path and accelerates along the valley floor, exactly as in the two worked examples.

Momentum's exponentially weighted average of past gradients behaves like the running averages shown in the explorer above: recent values dominate, older ones fade geometrically, and the result is smoother than any single noisy reading.

What this means in practice

Momentum is close to a free upgrade over plain SGD for deep networks — it's cheap to compute, requires storing one extra vector per parameter, and reliably speeds convergence on the elongated, ravine-shaped loss surfaces typical of deep learning. Nesterov's look-ahead correction usually helps a bit further, especially near a minimum where uncorrected momentum tends to overshoot.