The One-Cycle Policy and Superconvergence

The intuitive learning rate schedule is monotonic decay: start reasonably high, shrink it throughout training as the model settles in. That works, but it leaves a surprising amount of speed on the table. Leslie Smith found that deliberately pushing the learning rate much higher than the conventional wisdom would suggest — for a bounded window in the middle of training, then bringing it back down — can reach the same or better accuracy in a fraction of the epochs, a phenomenon he called superconvergence. The one-cycle policy is the schedule that produces it.

The analogy: a single deliberate overshoot, not a wild swing

Think of tuning a piano string: it's tightened past the target pitch briefly, on purpose, then eased back down to settle exactly where it needs to be — deliberately overshooting produces a more stable, accurate final tension than approaching the target only from below. The one-cycle policy applies the same logic to training: a learning rate that's briefly higher than "safe" pushes the model through flatter, more robust regions of the loss landscape that a purely decaying schedule would have crept past too cautiously to ever reach.

The schedule

One cycle has three phases across the total training steps $T$, split roughly 40/40/20 in Smith's original recipe, between a low rate $\eta_{\min}$ and a peak $\eta_{\max}$ (often 3–10× the rate a conventional schedule would use):

$$\eta(t) = \begin{cases} \eta_{\min} + (\eta_{\max}-\eta_{\min})\dfrac{t}{0.4T} & 0 \le t < 0.4T \quad \text{(rising)} \\[6pt] \eta_{\max} - (\eta_{\max}-\eta_{\min})\dfrac{t - 0.4T}{0.4T} & 0.4T \le t < 0.8T \quad \text{(falling back down)} \\[6pt] \eta_{\min} \cdot \text{(further, smaller decay)} & 0.8T \le t \le T \quad \text{(fine-tune)} \end{cases}$$

In words: the learning rate climbs linearly from a low value to a high peak over the first 40% of training, comes back down to roughly where it started over the next 40%, then anneals to a very small value for the final stretch, letting the model settle into a precise minimum once the aggressive exploration phase is over. Momentum is typically cycled in the opposite direction — low while the learning rate is high, high while the learning rate is low — since a large step and a large momentum multiplier together would compound into an excessively large update.

Worked example 1: computing the schedule at three checkpoints

$T = 1000$ steps, $\eta_{\min} = 0.001$, $\eta_{\max} = 0.01$. At $t=200$ (rising phase): $\eta = 0.001 + (0.009)(200/400) = 0.0055$. At $t=400$ (peak): $\eta = 0.01$, matching $\eta_{\max}$. At $t=600$ (falling phase): $\eta = 0.01 - 0.009 \times (200/400) = 0.0055$ — symmetric with $t=200$, as rise and fall mirror each other in this linear version.

Worked example 2: epochs saved, a documented case

Smith's original experiments trained a ResNet on CIFAR-10 to a fixed target accuracy: a conventional decaying schedule needed roughly 80 epochs; a one-cycle schedule with a substantially higher peak rate reached the same accuracy in about 8–15 epochs — a factor of roughly 5–10× fewer epochs for comparable final accuracy. The mechanism proposed is that the high-rate phase acts as a regularizer, pushing the optimizer out of narrow, sharp minima toward flatter regions associated with better generalization, and the final low-rate phase then converges precisely within that better region.

What this means in practice

torch.optim.lr_scheduler.OneCycleLR implements this directly and is a strong default when training time or compute budget is the binding constraint. Finding the right $\eta_{\max}$ still requires the same learning-rate-range test Smith describes (increase the rate exponentially over a short run and watch where loss starts climbing) — an $\eta_{\max}$ too aggressive for one-cycle can still diverge, so the peak rate is found empirically per model and dataset, not a fixed multiplier.