The One-Cycle Learning-Rate Policy

The learning rate controls how big a step a model's parameters take with each round of training. Too small, and training crawls along, needing far more iterations than necessary to reach a good result. Too large, and training can bounce around erratically or diverge outright. The obvious fix is to pick something in the middle and hold it fixed, or slowly decay it — but the one-cycle policy found a better approach almost by accident: deliberately push the learning rate much higher than the usual "safe" value for part of training, on a specific schedule, and training often converges faster and to a better result than with a conventional constant or steadily decaying rate.

The schedule has two phases within a single run. First, the learning rate ramps up from a small starting value to a peak that's often far higher than what would normally be considered safe — high enough that using it constantly would risk instability. Then it ramps back down, usually to a value even smaller than where it started, for the final stretch of training. The idea is that the high-rate middle phase acts like a controlled, temporary "shake" that helps the model jump over shallow, poor local regions of the loss surface early, while the low-rate final phase lets it settle precisely into a good solution once it's already in a good region.

Picture training a model for 100 epochs. Under one-cycle, the learning rate might climb from 0.001 up to 0.1 over the first 45 epochs, then descend from 0.1 down to 0.0001 over the remaining 55 — never sitting at a single moderate value the way a constant-rate schedule would, and reaching a comparable or better result in fewer total epochs than gradually decaying from a smaller starting point.

What this means in practice

One-cycle schedules are widely used for training deep learning models, including models used for return prediction or feature extraction in quant research, precisely because they tend to cut training time substantially compared to naive constant or step-decay schedules. The main practical cost is that the peak learning rate and the length of the cycle are extra hyperparameters that need to be tuned per problem — get the peak too high and training genuinely destabilizes rather than being helped by the "shake."