Gradient Clipping

Training a neural network means repeatedly nudging its weights in the direction that reduces the loss, and the size of each nudge is the gradient. Occasionally — especially in deep networks or recurrent networks processing long sequences — the gradient computed from a particular batch is enormous, sometimes by orders of magnitude, because errors compound as they propagate backward through many layers or time steps. Taking that update literally can throw the weights somewhere useless, and training can diverge or the loss can spike to NaN.

Gradient clipping is the fix: before applying an update, check the size of the gradient and, if it exceeds some threshold, rescale it back down to that threshold while keeping its direction unchanged. The most common version, clipping by norm, computes the overall magnitude of the gradient vector and if it's larger than a chosen cap (say, 1.0 or 5.0), multiplies the whole vector by cap-divided-by-magnitude so the rescaled gradient has exactly the cap's length. A cruder alternative, clipping by value, simply caps each individual gradient component independently.

In practice this shows up as one line in a training loop, applied right before the optimizer step, and it is close to a default setting for training RNNs, LSTMs, and transformers, where occasional gradient spikes are common. It doesn't fix a genuinely broken model or learning rate that's too high on average — it only stops the rare catastrophic spike from derailing an otherwise sound training run.