Loss Scaling and FP16 Overflow

Training in 16-bit floating point (FP16) roughly halves memory use and can double throughput compared to 32-bit, but FP16 has a narrow exponent range: any value smaller than about $6\times10^{-5}$ in magnitude rounds straight to zero. Many gradients in a deep network, especially in early layers or late in training, are naturally that small, and once a gradient underflows to zero it teaches the model nothing at all on that step — silently, with no error raised.

Loss scaling works around this without touching the numbers that actually matter. Before running the backward pass, the loss is multiplied by a large constant (a common starting point is 1024, sometimes adjusted automatically), which by the chain rule multiplies every gradient by that same factor, pushing the small gradients back up into FP16's representable range instead of collapsing to zero. Immediately before the optimizer applies the update, the gradients are divided back down by that same factor, restoring their true scale. Weights themselves are typically still updated in FP32 to avoid accumulating rounding error over many steps, with FP16 used only for the compute-heavy forward and backward passes. Choosing the scaling factor is itself a small balancing act: too small and gradients still underflow; too large and gradients that were already a healthy size now overflow FP16's upper limit and turn into infinities or NaNs. Modern frameworks handle this with dynamic loss scaling: start with a large factor, watch for overflow after each step, and halve the factor whenever one occurs, gradually increasing it again after a run of clean steps, so the scale tracks whatever range the gradients actually need without a person having to tune it by hand. Because dynamic loss scaling requires almost no manual configuration and adds negligible overhead, it is now the default behind most "just enable FP16" flags in mainstream training frameworks.