Mixed-Precision Training

Every number a neural network stores — a weight, an activation, a gradient — lives in some fixed number of bits, and that choice is a lever nobody outside systems engineering usually pulls. The default, 32-bit floating point, gives huge dynamic range and fine precision, but it costs twice the memory and twice the compute of its 16-bit cousin for every single multiplication in a network with billions of them. The obvious idea — just use 16-bit everywhere and train twice as fast — turns out to break training outright unless you're careful about exactly where the smaller format is safe to use.

The analogy: a tape measure with too few markings

Standard 32-bit precision is a tape measure with fine gradations — you can read off a length like $52.31847$ millimetres to five decimal places. Switch to 16-bit and you get a coarser tape, gradations further apart, and any true length that falls between two marks gets rounded to the nearest one. For most measurements in a building, that coarseness costs you nothing. But if you're trying to measure a change of $0.0001$ millimetres against a coarse tape whose smallest mark is $0.01$, the change simply disappears — it rounds to zero. Many of the tiny numbers a neural network needs to track during training, especially gradients multiplied by a small learning rate, are exactly that kind of small change, and a naive switch to 16-bit rounds a meaningful chunk of them straight to zero, which silently stalls learning.

The recipe: two formats, one safety net

Mixed precision keeps weights in 16-bit format (FP16) for the matrix multiplications that dominate compute time, but keeps a master copy of the weights and the loss computation in 32-bit (FP32) where precision actually matters. Two techniques carry the weight:

Loss scaling. Multiply the loss by a constant $S$ before backpropagating, then divide the resulting gradients by the same $S$ before the weight update:

$$\tilde{g} = \frac{1}{S}\,\nabla_\theta\,(S \cdot L(\theta))$$

In words: gradients in FP16 that are too small to represent get pushed up into FP16's representable range by scaling the loss up first, computed as usual, then scaled back down to their true size afterward — the scaling and unscaling cancel out mathematically, so the update is unaffected, but the intermediate numbers never fall below FP16's precision floor. $S$ is typically a large power of two, like $2^{14}$, chosen (often automatically, adjusted upward when no overflow occurs and downward when one does) so scaled gradients land comfortably inside FP16's range without overflowing it.

Master weights. Keep one full-precision (FP32) copy of every weight purely for the update step, even though the forward and backward passes run in FP16. A weight update is often a very small FP16 number added to a much larger one, and FP16 alone would round tiny, cumulative updates away entirely; storing the master copy in FP32 lets those small updates actually accumulate over thousands of steps instead of vanishing.

Worked example 1: why loss scaling is needed, with real numbers

Suppose a particular gradient value is $0.00003$. FP16's smallest normal representable magnitude is roughly $6.1 \times 10^{-5}$ — anything smaller than that either rounds to zero or loses most of its precision as a "subnormal" number. So $0.00003 = 3 \times 10^{-5}$ falls below the safe threshold and would be stored inaccurately or as zero, meaning that weight simply stops updating from this gradient.

Scale the loss by $S = 1024$ before backpropagating. The gradient scales proportionally too: $0.00003 \times 1024 = 0.0307$. That value sits comfortably within FP16's precise range and survives the backward pass intact. After computing it, divide by the same $1024$ to recover the true, unscaled gradient of $0.0307 / 1024 = 0.00003$ before applying it to the FP32 master weight. The scaling never changes the mathematical answer — it only moves the number to a range the format can actually hold onto during the computation.

Worked example 2: memory and speed, back-of-envelope

A model has 175 million parameters. Stored purely in FP32 (4 bytes each), just the weights take $175{,}000{,}000 \times 4 = 700{,}000{,}000$ bytes $\approx 700$ MB. Training also needs gradients and, for a common optimiser, two additional per-parameter state buffers (Adam's momentum and variance terms) — four FP32 buffers total, so roughly $4 \times 700 = 2{,}800$ MB just for these, before activations.

Under mixed precision, the FP16 forward/backward copies of weights and gradients cost half as much per element: $175{,}000{,}000 \times 2 = 350{,}000{,}000$ bytes $\approx 350$ MB each. The FP32 master weights and optimiser state stay full-size, so total memory isn't a clean half, but the FP16 activations — which scale with batch size and sequence length and often dominate memory in practice — do roughly halve. Combined with FP16 matrix multiplications running roughly twice as fast on hardware with dedicated FP16 cores (common on modern GPUs), a well-implemented mixed-precision setup typically trains 1.5 to 3 times faster with a meaningfully smaller memory footprint, for the same model and the same final accuracy.

What this means in practice

Mixed precision is close to a default in modern deep learning frameworks now (PyTorch's autocast, TensorFlow's automatic mixed precision), and it usually requires only a few lines of code to enable. Where it needs attention: loss scaling must be monitored, since too aggressive a scale causes gradient overflow (values that saturate to infinity in FP16), which most frameworks detect automatically and respond to by skipping that step and halving the scale. Certain operations — softmax, layer normalization, loss reductions — are kept in FP32 even in a "mixed precision" run because they're numerically sensitive to the reduced range. In quant research specifically, where models are frequently retrained on rolling windows and training throughput directly limits how much hyperparameter search or how many backtests you can run in a day, the roughly 2x speedup is often the difference between a research idea that gets properly explored and one that doesn't get tried at all because a single run takes too long.

The gap between FP16 and FP32's representable range grows exponentially as numbers get larger — this explorer's exponential curve is the same shape as the range FP16 loses relative to FP32 at the extremes, which is exactly why loss scaling exists: to keep gradients away from the region where that gap actually bites.