The Linear Scaling Rule and Learning-Rate Warmup

Training a model faster usually means processing more examples per step — a bigger batch. But if you double the batch size and keep everything else fixed, each step now averages the gradient over twice as many examples, which makes that averaged gradient less noisy and, on average, points in a more reliable direction without shrinking in size. If you don't also adjust the learning rate, you take the same size step using a more confident-but-not-smaller signal roughly half as often per epoch — training slows down relative to what the bigger batch could support. The fix sounds simple: scale the learning rate up with the batch size. The trap is doing that from step one.

The analogy: a convoy learning to ride together

Picture a single cyclist finding a route by feel, adjusting course after every pothole. Now put eight cyclists on a tandem-style convoy, all pedalling together — their combined sense of the road is steadier, so they can afford to turn the handlebars more aggressively per correction and still not zig-zag off the path. That is the case for a bigger learning rate with a bigger batch. But if that whole convoy slams into a hard turn at full speed before anyone has found their balance, it doesn't ride smoothly — it wobbles and can tip over. So they start slow, feel out the road together, and only ease up to full turning speed once they are stable. That easing-in period is warmup.

The rule and the ramp

The linear scaling rule says: if you multiply the batch size by $k$, multiply the learning rate by $k$ too.

$$\eta_{\text{new}} = \eta_{\text{base}} \times \frac{B_{\text{new}}}{B_{\text{base}}}$$

In words: the new learning rate ($\eta_{\text{new}}$) equals the base rate ($\eta_{\text{base}}$) scaled by how many times bigger the new batch ($B_{\text{new}}$) is than the base batch ($B_{\text{base}}$). It works because the noise in the gradient estimate shrinks roughly with the square root of batch size, so the step size can grow with batch size while keeping a similar effective amount of "wander" per epoch.

Warmup delays reaching that full scaled-up rate. Instead of starting training at $\eta_{\text{new}}$, the learning rate ramps up linearly (or on some other schedule) from a small value, or zero, over the first few hundred to few thousand steps, only reaching $\eta_{\text{new}}$ once the model's weights and any batch-statistics have had time to settle out of their random initial state.

Worked example 1: scaling the rate

A model trains well with batch size $B_{\text{base}} = 256$ and learning rate $\eta_{\text{base}} = 0.1$. A larger GPU cluster lets you use batch size $B_{\text{new}} = 2048$ — eight times bigger. By the linear scaling rule:

$$\eta_{\text{new}} = 0.1 \times \frac{2048}{256} = 0.1 \times 8 = 0.8$$

Jumping straight to a learning rate of $0.8$ at step one, on a model whose weights are still near their random initial values, is exactly the wobbling-convoy scenario — large, confident steps taken before the model has any sense of the loss surface's shape.

Worked example 2: the warmup ramp

Suppose warmup is linear over the first 5 epochs, from $0$ up to the target $\eta_{\text{new}} = 0.8$. At epoch $e$ (out of 5), the learning rate is:

$$\eta(e) = 0.8 \times \frac{e}{5}$$

At epoch 1: $\eta = 0.8 \times 1/5 = 0.16$. At epoch 2: $\eta = 0.8 \times 2/5 = 0.32$. At epoch 5: $\eta = 0.8 \times 5/5 = 0.8$, the full scaled rate, reached only once training has had four epochs to stabilize.

Drag the learning rate up in the explorer above and watch a single large step overshoot the loss curve's minimum and bounce away, versus a smaller step that crawls in steadily — warmup exists precisely to avoid that early overshoot while the model is at its most fragile.

What this means in practice

Every large-batch training recipe — image classifiers trained on hundreds of GPUs, large language models trained on thousands — uses some version of this pairing: scale the rate with the batch, but never apply the scaled rate immediately. Skipping warmup on a large-batch, large-learning-rate run is one of the most common causes of a loss that spikes or diverges in the first few hundred steps, long before any other part of the recipe could be blamed.

Practice

1. Base setup: batch 128, learning rate 0.05. You move to batch 1024. What is the linearly scaled learning rate?
2. Warmup is linear over 4 epochs to a target of 0.4. What is the rate at epoch 3?
3. Why does gradient noise shrinking with batch size make a larger learning rate tolerable, in one sentence?