Training Instability and Loss Spikes

A model can train smoothly, loss falling week after week, then suddenly spike: the loss value jumps up sharply within a handful of steps, sometimes recovering on its own, sometimes never recovering and instead diverging to a much worse point or even to numerical garbage (NaN). For a training run costing weeks of compute time, a spike that doesn't recover can be enormously expensive, so understanding why they happen — and building automatic responses to them — is an essential, unglamorous part of running training at scale.

The analogy: a truck hitting a pothole at speed

A truck cruising down a smooth highway can hit one unexpected pothole and lurch — the impact is disproportionate to how smooth everything was a second earlier. Usually the suspension absorbs the jolt. But sometimes the jolt throws something loose, and the truck drives worse even after the road smooths out. A loss spike works the same way: one unusually bad batch, or an unlucky combination of large gradients, jolts the optimizer — usually absorbed, but sometimes bad enough to knock the model into a worse region it can't easily climb back out of.

Where spikes actually come from

A handful of concrete mechanisms drive most instability, and they compound: a learning rate that is too high for the current point in training produces oversized steps whenever the gradient happens to be unusually large. Gradient magnitude, not gradient direction, is the direct culprit — even a correct direction, multiplied by a huge magnitude and a full-size learning rate, overshoots badly:

$$\theta_{t+1} = \theta_t - \eta\, g_t$$

In words: the update is the learning rate ($\eta$) times the gradient ($g_t$); if $g_t$ is unusually large — from a rare batch, from certain layers early in training before their scale has settled, or from numerical issues in reduced-precision arithmetic — the update $\eta g_t$ can be enormous even though $\eta$ never changed. Reduced-precision training (16-bit numbers, to save memory) compounds this: values that would be fine in full precision can overflow, injecting numerical noise right where gradients are already large. In large distributed jobs, a single corrupted batch reaching one worker can inject a wrong gradient into the shared all-reduced update, propagating the damage to every worker at once.

Worked example 1: gradient clipping catches an outlier step

Suppose gradients across recent steps have had norm (overall magnitude) around $2$, but one step produces a gradient with norm $50$ — a rare bad batch. With learning rate $\eta = 0.01$ and no clipping, the update size scales with $50$, twenty-five times the typical step. Gradient clipping caps the norm at, say, $5$ before applying it: the gradient vector is rescaled by $5/50 = 0.1$, so its effective norm becomes exactly $5$ instead of $50$ — the direction is preserved, but the magnitude is capped to something close to what training has been seeing, preventing the huge, destabilizing update while still moving in a useful direction.

Worked example 2: rolling back to a checkpoint after a spike

A team monitors loss and sets an automated rule: if loss at step $t$ exceeds $1.5\times$ the trailing 100-step average, halt training. Suppose the trailing average is $2.0$ and a spike takes the loss to $2.0 \times 1.5 = 3.0$ or higher, triggering the rule at, say, step 48,200. The response: reload the last saved checkpoint (say, step 48,000), skip forward past the batch of data that caused the issue (isolated from the data pipeline's shuffled order), and resume from there, sometimes with a temporarily lower learning rate for the next several hundred steps as an extra safety margin. Losing 200 steps of progress is far cheaper than losing the whole run to an unrecovered divergence.

What this means in practice

Large training runs routinely use gradient clipping as a default safety measure, monitor loss automatically for spikes rather than relying on a human watching a dashboard, and checkpoint frequently enough that a rollback loses minimal progress. Some spikes are genuinely mysterious even to experienced teams — large language model training reports have documented spikes with no clearly identified single cause, resolved only by rolling back and skipping the suspect data batch.

Practice

1. A gradient has norm 30 and the clip threshold is 6. What rescaling factor is applied, and what is the resulting norm?
2. Why can a spike that "recovers on its own" still leave the model in a worse spot than before?
3. Why does reduced-precision (16-bit) training make spikes more likely, in one sentence?