Learning Rate Warmup

A network's weights start out random, which means its earliest gradients are computed from a model that has essentially no idea what it's doing — those gradients can point in wildly unreliable directions, or have unusually large magnitude, simply because nothing has stabilized yet. Take a full-sized learning rate step in that direction immediately, and the resulting weight update can be large enough to push the network into a genuinely bad region it then struggles to recover from — sometimes visible as loss spiking or even diverging to NaN in the very first few hundred steps. Warmup avoids this by starting the learning rate at (or near) zero and ramping it up over the first portion of training.

The analogy: warming up a car engine before flooring it

Cold engine oil is thick and hasn't reached the gaps it needs to lubricate yet — flooring the accelerator immediately puts real stress on parts that aren't ready for it. A driver who idles for a minute, then gradually opens the throttle, lets the engine reach a state where full power is safe to apply. Learning rate warmup is the same idea applied to training: the "engine" (the randomly initialized network) needs a short period of gentle, small updates before it's in a state where the large, efficient updates of the full learning rate are safe rather than destabilizing.

The formula

A common linear warmup schedule, over the first $T_w$ steps, with target (post-warmup) learning rate $\eta_{\max}$:

$$\eta(t) = \eta_{\max} \cdot \frac{t}{T_w}, \qquad t = 1, \dots, T_w$$

In words: the learning rate at step $t$ is a simple fraction of the target rate, growing linearly from nearly 0 at $t=1$ up to the full $\eta_{\max}$ at $t = T_w$; after step $T_w$, training proceeds with whatever schedule (constant, cosine decay, step decay) was planned for the rest of training, now starting from a network that has already taken thousands of small, stabilizing steps.

Worked example 1: computing the warmup schedule by hand

Target learning rate $\eta_{\max} = 0.001$, warmup length $T_w = 1000$ steps. At $t=1$: $\eta = 0.001 \times 1/1000 = 0.000001$ — essentially a no-op. At $t = 500$: $\eta = 0.0005$, half the target rate. At $t = 1000$: $\eta = 0.001$, the full rate is reached, and the schedule hands off to whatever decay policy governs the remaining, much longer main phase.

Worked example 2: why large-batch training needs longer warmup

A larger batch size reduces gradient noise per step, which typically justifies a proportionally larger target learning rate (the "linear scaling rule": doubling batch size roughly doubles $\eta_{\max}$). But a larger $\eta_{\max}$ makes the early unstable-gradient period more dangerous, since the destabilizing step gets bigger too. A well-known large-batch ImageNet recipe found that scaling batch size 32-fold required extending warmup several-fold, or training diverged outright regardless of how carefully everything else was tuned — warmup length itself became a hyperparameter that had to scale alongside batch size.

The explorer below shows optimization steps on a loss surface — watch how a large fixed step size from an unstabilized start can overshoot and bounce, exactly the failure warmup prevents.

What this means in practice

Warmup is close to mandatory for training transformers from scratch and for any large-batch run, appearing by default in most modern recipes (a few hundred to a few thousand steps, scaled with batch and model size); Adam and other adaptive optimizers benefit too, since their per-parameter variance estimates are unreliable in the first handful of steps. A quant fine-tuning a pretrained model on a smaller financial dataset generally needs a much shorter warmup, since the weights already sit in a reasonable region.