Learning Rate Schedules

A single, fixed learning rate faces a genuine dilemma: large enough to make fast progress early in training, but small enough not to bounce wildly around the minimum once training gets close to it, are usually two different numbers. A learning rate that's perfect for the first hundred steps is often too large for the last hundred, and one that's safe for the last hundred would have crawled agonisingly slowly through the first. A schedule resolves the dilemma by not picking one number at all — it changes the rate as training proceeds.

The analogy: driving toward a parking spot

Driving across a large empty parking lot toward a spot far away, you'd naturally go fast — there's plenty of room and no need for caution. As you get close to the actual spot, you slow down, because now precision matters more than speed and a fast approach risks overshooting or clipping something. Nobody drives the entire trip, start to finish, at one single fixed speed chosen as some compromise between "fast enough to get there" and "slow enough to park precisely." A learning rate schedule is this same instinct applied to training: cover distance fast early, then ease off as you approach something you don't want to overshoot.

The maths: a few common shapes

Step decay multiplies the rate by a fixed factor every so many steps:

$$\eta_t = \eta_0 \cdot \gamma^{\lfloor t / s \rfloor}$$

Here $\eta_0$ is the starting rate, $\gamma<1$ the decay factor, $s$ the number of steps between drops, and $\lfloor \cdot \rfloor$ rounds down. In words: every $s$ steps, multiply the current rate by $\gamma$ — the rate drops in discrete jumps, like halving every 10 epochs.

Cosine annealing decays smoothly, following one quarter of a cosine wave down from $\eta_0$ to near zero over $T$ total steps:

$$\eta_t = \eta_0 \cdot \frac{1}{2}\left(1 + \cos\left(\frac{\pi t}{T}\right)\right)$$

In words: the rate starts at $\eta_0$, barely changes at first, then falls steeply through the middle of training, then flattens out again as it approaches zero near the end — a smooth version of the same "fast then careful" idea, without step decay's abrupt jumps.

Warmup does the opposite at the very start: ramp the rate up linearly from zero over the first few hundred steps, before any decay schedule takes over, because a large rate applied to randomly initialised weights before the network has found any sensible direction can cause an early, destabilising blowup.

Worked example 1: step decay, three drops

$\eta_0 = 0.1$, $\gamma = 0.5$, drop every $s=1000$ steps. At step $500$: $\lfloor 500/1000\rfloor = 0$, so $\eta = 0.1 \times 0.5^0 = 0.1$. At step $1500$: $\lfloor 1500/1000 \rfloor = 1$, so $\eta = 0.1 \times 0.5 = 0.05$. At step $3200$: $\lfloor 3200/1000\rfloor=3$, so $\eta = 0.1 \times 0.5^3 = 0.0125$. Each threshold crossed halves the rate again — by step 3200 the effective rate is an eighth of where it started, even though nothing about the loss triggered the drop, only the step count.

Worked example 2: cosine annealing at three points

$\eta_0 = 0.01$, $T = 1000$ total steps. At $t=0$: $\eta = 0.01 \times 0.5(1+\cos 0) = 0.01 \times 1 = 0.01$ — full rate. At $t=500$ (halfway): $\eta = 0.01 \times 0.5(1+\cos(\pi/2)) = 0.01\times0.5(1+0)=0.005$ — exactly half. At $t=1000$ (the end): $\eta = 0.01\times0.5(1+\cos\pi) = 0.01\times0.5(1-1)=0$ — the rate has smoothly reached zero right as training ends, so the very last steps are essentially fine-tuning with an infinitesimal nudge, which is exactly the "precise parking" behaviour the schedule was designed for.

Picture the cosine schedule's shape — high and flat at the start, a steep fall through the middle, flat near zero at the end — a curve of that general falling character to the one you can explore here, just built from cosine instead of a power law.

The explorer above shows an estimate needing large early corrections and only small late ones to settle — that's the same logic a decaying schedule imposes deliberately: big steps while far from the answer, tiny ones while refining it.

What this means in practice

Cosine annealing and warmup are close to standard in modern deep learning training recipes, especially for large models where a bad early learning-rate spike can waste enormous compute before it's caught. Adaptive optimisers like Adam already adjust per-parameter rates, but they don't remove the value of also scheduling the global rate over time — the two operate on different axes and are commonly combined.