Cosine Annealing and Warm Restarts

A learning rate that stays constant throughout training tends to keep bouncing around near the bottom of the loss surface without ever settling precisely — like trying to park a car by only ever using full acceleration or full brake. Dropping the rate down helps it settle, but a schedule that drops it in sudden steps (say, cutting it by 10x every 30 epochs) creates jarring transitions: training happily cruising at one rate, then abruptly forced into much smaller steps. Cosine annealing smooths that transition into a gentle curve, and warm restarts add a deliberate way to escape a valley the smooth descent settled into too early.

The analogy: coasting a bike to a stop, then pedalling off again

Braking a bicycle in one sudden clamp is jarring; easing off the pedals and letting the bike gradually decelerate to a stop is smooth — that gradual glide, fast at first and slower as you approach zero, is the shape of a cosine curve over its first half. Cosine annealing lets the learning rate glide down exactly like that: fast early, slowing as it nears its minimum. A warm restart is choosing to hop back on and pedal hard again once you've coasted to a stop at some spot, in case a bit further down the road there's a better spot to end up — rather than assuming the first place you coasted to a stop was the only worthwhile one.

The shape of the curve

Cosine annealing sets the learning rate at step $t$ (within a cycle of length $T$) as:

$$\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max}-\eta_{\min})\left(1+\cos\!\left(\frac{t}{T}\pi\right)\right)$$

In words: the rate starts at $\eta_{\max}$ when $t=0$ (since $\cos(0)=1$), glides smoothly down following a cosine curve, and reaches $\eta_{\min}$ at $t=T$ (since $\cos(\pi)=-1$). Unlike a step schedule, the rate of decrease itself changes smoothly — fastest around the midpoint of the cycle, gentlest near both ends — rather than jumping discontinuously.

A warm restart resets $t$ back to $0$ once a cycle finishes, sending the learning rate immediately back up to $\eta_{\max}$ and starting a fresh cosine descent, often with the cycle length $T$ doubled each time (a schedule called SGDR). The idea: once the optimizer has settled somewhere by the end of one cosine cycle, a sudden jump back to a high learning rate can kick it out of that spot and let it explore for a potentially better one, while the eventual re-descent lets it settle again.

Worked example 1: one cosine cycle

Let $\eta_{\max}=0.1$, $\eta_{\min}=0$, cycle length $T=10$. At $t=0$: $\eta_0 = 0 + 0.05(1+\cos(0)) = 0.05(2)=0.1$. At $t=5$ (halfway): $\cos(\pi/2)=0$, so $\eta_5 = 0.05(1+0)=0.05$ — exactly half the max rate, at the point of steepest descent. At $t=10$ (cycle end): $\cos(\pi)=-1$, so $\eta_{10}=0.05(1-1)=0$. The rate glides from $0.1$ to $0.05$ to $0$ smoothly across the cycle, slowing its own descent as it approaches zero.

Worked example 2: a warm restart

Continuing that setup with SGDR-style doubling: right after $t=10$, the rate resets to $\eta_{\max}=0.1$ and a new cycle of length $T=20$ begins (double the first). At the new cycle's midpoint, $t=10$ into the new cycle: $\eta = 0.05(1+\cos(\pi/2))=0.05$, same relative shape, but now stretched over twice as many steps, giving the optimizer more time to explore and settle in this longer second cycle before the next restart.

What this means in practice

Cosine annealing (often without restarts) is a common default schedule for training modern deep networks because the smooth descent, especially the slow final approach to a low rate, tends to land in a well-settled minimum without the jarring transitions of step schedules. Warm restarts are also useful for generating a series of good, somewhat-different weight snapshots at each cycle's end — a natural feeder into the weight-averaging techniques used in snapshot ensembling and Stochastic Weight Averaging.

Practice

1. With $\eta_{\max}=0.2$, $\eta_{\min}=0.01$, $T=8$, what is $\eta_t$ at $t=4$?
2. Why does a step-schedule's sudden rate drop create a more jarring transition than cosine annealing's?
3. What is one reason you might want the snapshot right before a warm restart rather than right after?