The One-Cycle Policy and Superconvergence

The intuitive learning rate schedule is monotonic decay: start reasonably high, shrink it throughout training as the model settles in. That works, but it leaves a surprising amount of speed on the table. Leslie Smith found that deliberately pushing the learning rate much higher than the conventional wisdom would suggest, for a bounded window in the middle of training, then bringing it back down, can reach the same or better accuracy in a fraction of the epochs, a phenomenon he called superconvergence. The one-cycle policy is the schedule that produces it.

The analogy: a single deliberate overshoot, not a wild swing

Think of tuning a piano string: it's tightened past the target pitch briefly, on purpose, then eased back down to settle exactly where it needs to be, deliberately overshooting produces a more stable, accurate final tension than approaching the target only from below. The one-cycle policy applies the same logic to training: a learning rate that's briefly higher than "safe" pushes the model through flatter, more robust regions of the loss landscape that a purely decaying schedule would have crept past too cautiously to ever reach.

The schedule

One cycle has three phases across the total training steps $T$, split roughly 40/40/20 in Smith's original recipe, between a low rate $\eta_{\min}$ and a peak $\eta_{\max}$ (often 3–10× the rate a conventional schedule would use):

In words: the learning rate climbs linearly from a low value to a high peak over the first 40% of training, comes back down to roughly where it started over the next 40%, then anneals to a very small value for the final stretch, letting the model settle into a precise minimum once the aggressive exploration phase is over. Momentum is typically cycled in the opposite direction, low while the learning rate is high, high while the learning rate is low, since a large step and a large momentum multiplier together would compound into an excessively large update.

Worked example 1: computing the schedule at three checkpoints

$T = 1000$ steps, $\eta_{\min} = 0.001$, $\eta_{\max} = 0.01$. At $t=200$ (rising phase): $\eta = 0.001 + (0.009)(200/400) = 0.0055$. At $t=400$ (peak): $\eta = 0.01$, matching $\eta_{\max}$. At $t=600$ (falling phase): $\eta = 0.01 - 0.009 \times (200/400) = 0.0055$, symmetric with $t=200$, as rise and fall mirror each other in this linear version.

Worked example 2: epochs saved, a documented case

Smith's original experiments trained a ResNet on CIFAR-10 to a fixed target accuracy: a conventional decaying schedule needed roughly 80 epochs; a one-cycle schedule with a substantially higher peak rate reached the same accuracy in about 8–15 epochs, a factor of roughly 5–10× fewer epochs for comparable final accuracy. The mechanism proposed is that the high-rate phase acts as a regularizer, pushing the optimizer out of narrow, sharp minima toward flatter regions associated with better generalization, and the final low-rate phase then converges precisely within that better region.

What this means in practice

torch.optim.lr_scheduler.OneCycleLR implements this directly and is a strong default when training time or compute budget is the binding constraint. Finding the right $\eta_{\max}$ still requires the same learning-rate-range test Smith describes (increase the rate exponentially over a short run and watch where loss starts climbing), an $\eta_{\max}$ too aggressive for one-cycle can still diverge, so the peak rate is found empirically per model and dataset, not a fixed multiplier.