Transformer Training Instabilities and Loss Spikes

Training a large language model can run smoothly for weeks, loss decreasing steadily, and then — with no change in data, code, or hyperparameters — the loss suddenly jumps upward, sometimes sharply, before either recovering on its own or degrading the model permanently. These loss spikes are among the most expensive problems in large-scale training, precisely because they tend to appear only at the scale of tens or hundreds of billions of parameters, where a single run costs a great deal and a failed one cannot simply be discarded. Understanding why they happen is distinct from ordinary small-scale hyperparameter tuning, because the failure modes often don't show up at smaller scale at all.

The analogy: a truck fine on flat roads, unstable on a steep unexpected grade

A delivery truck can drive smoothly for hundreds of miles using a fixed set of gears and speeds. But hit one unusually steep, cambered hill, and the same strategy that worked everywhere else can suddenly cause the load to shift — not because the driver changed anything, but because that stretch of road exposed a weakness flat roads never triggered. Large-model training is similar: an optimiser configuration that has run stably for millions of steps can hit a particular batch, or numerical configuration, that exposes an instability the vast majority of training never encounters.

Where spikes come from

Several distinct mechanisms are implicated, and real spikes are often a combination:

Gradient and activation scale blow-ups. Deep networks can, at certain points in training, produce activations or gradients whose magnitude grows sharply across layers. With Adam, a burst of unusually large gradients can distort its running estimates of gradient mean and variance in a way that persists for many steps, compounding the damage rather than self-correcting.

$$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$$

In words: Adam keeps a running average of recent gradients $m_t$ and their squared magnitudes $v_t$; one anomalously large gradient $g_t$ pollutes both, and since $v_t$ sits in the update's denominator, that distortion can make subsequent updates unexpectedly large or erratic for many steps, not just the one bad step.

Data artefacts. A batch with an unusual sequence — heavily repeated tokens, degenerate formatting, an outlier document — produces an unusually large loss and gradient, and at large batch sizes and learning rates the resulting update can push weights into a worse region.

Precision and numerical issues. Reduced precision (bfloat16/float16) trades numerical range for speed; certain layer configurations (see Pre-Norm vs Post-Norm Transformer Blocks) are more prone to values overflowing at exactly the wrong moment.

Worked example 1: reading a loss curve

A training log shows loss declining from 3.2 to 2.4 over 40,000 steps, then abruptly jumping to 3.8 at step 40,200, before slowly declining again over the next 2,000 steps back toward the pre-spike trend. This shape — sudden jump, partial or full self-recovery over a much longer window — is the classic signature of a transient instability; a genuinely broken run instead shows loss staying elevated, or diverging (often visible as NaN) with no recovery at all.

Worked example 2: a mitigation and its effect

A team observes recurring spikes and applies gradient clipping — rescaling the gradient vector so its norm never exceeds a threshold $c$ (commonly ~1.0):

$$g' = g \cdot \min\!\left(1, \frac{c}{\lVert g \rVert}\right)$$

In words: if the gradient's magnitude $\lVert g \rVert$ already sits below $c$, leave it unchanged; if it exceeds $c$, shrink the vector so its magnitude equals $c$, preserving direction but capping size. Before clipping, one batch produced a gradient norm of 340 against a target step size calibrated for norms around 1–2 — an update roughly 200× larger than intended, plausibly the cause of a spike. After clipping at $c=1.0$, that same batch instead produces an update capped at the intended scale, and the spike does not occur.

Push the learning rate up in the explorer above until steps overshoot and bounce instead of settling — that visible divergence, at a small scale, is the same dynamic that produces a loss spike at full model scale.

What this means in practice

Production training runs at scale routinely save frequent checkpoints so that, when a spike is detected by automated monitoring, training can be rolled back to the last good checkpoint and resumed with a lower learning rate or a skipped batch, rather than risking permanent degradation. This operational discipline — not just the model architecture — is a large part of what makes multi-week training runs reliable in practice.

Practice

1. A gradient vector has norm 12, and the clipping threshold is $c=2$. What is the norm of the clipped gradient, and by what factor was the original gradient scaled down?
2. Why does polluting Adam's running variance estimate $v_t$ with one anomalously large gradient cause instability to persist for many steps afterward, rather than being corrected immediately at the next step?