Mixed Precision and bfloat16 Training

Every number in a neural network — weights, activations, gradients — is normally stored as a 32-bit floating-point number (fp32), which gives roughly 7 decimal digits of precision and a huge representable range. Most of that precision is never actually needed: a weight update that changes a value in its 6th decimal digit rarely changes the model's behavior in any meaningful way. Mixed precision training exploits this by doing the bulk of the arithmetic in a 16-bit format instead — half the memory, and roughly 2–8× the throughput on hardware built for it — while keeping a small number of precision-sensitive operations at full 32-bit accuracy.

The analogy: rounding a bill, keeping the ledger exact

A cashier can quote and compute change using prices rounded to the nearest cent all day without ever mis-balancing the till, provided the master ledger behind the scenes tracks every transaction to a finer precision so tiny rounding errors don't accumulate into a real discrepancy by the end of the day. Mixed precision training works the same way: the "storefront" arithmetic (most forward and backward computations) happens in cheap, lower-precision numbers, while a "master ledger" copy of the weights is kept in full 32-bit precision and is what actually gets updated — the low-precision copies are refreshed from it before each step, so rounding never compounds across the whole training run.

The two 16-bit formats, and why bfloat16 avoids one whole problem

Both fp16 and bfloat16 use 16 bits total, but split them differently between exponent bits (controlling range — how large or small a number can be) and mantissa bits (controlling precision — how many significant digits). fp16 uses 5 exponent bits and 10 mantissa bits: more precision, but a much narrower range than fp32, so small gradient values can underflow to exactly zero. bfloat16 uses 8 exponent bits (matching fp32 exactly) and only 7 mantissa bits: less precision per number, but the same representable range as fp32, so a value large or small enough to be a normal fp32 number essentially never overflows or underflows in bfloat16.

Worked example 1: why fp16 needs loss scaling and bfloat16 doesn't

A gradient value of $0.00003$ is well within fp32's range. fp16's smallest normal representable magnitude is around $6 \times 10^{-5}$ — this gradient underflows toward zero (or to a denormal value with almost no precision), silently vanishing before it can update the weights. The standard fix is loss scaling: multiply the loss by a large constant (say, 1024) before the backward pass, which scales every gradient up by the same factor — $0.00003 \times 1024 = 0.0307$, safely representable — then divide the weight update back down by 1024 afterward. bfloat16's exponent range matches fp32's directly ($10^{-38}$ to $10^{38}$, roughly), so $0.00003$ is stored without any special scaling step at all — this is the main practical reason bfloat16 has become the default over fp16 for large-model training.

Worked example 2: memory and throughput, concretely

A model with 1 billion parameters stored entirely in fp32 needs $1 \times 10^9 \times 4$ bytes $= 4$ GB just for the weights, plus roughly the same again for gradients and optimizer state (Adam keeps two extra moment buffers per parameter), pushing total memory well past 12 GB before activations are counted. Switching weights and gradients to bfloat16 (2 bytes each) while keeping an fp32 master copy for the optimizer update cuts forward/backward memory roughly in half and, on hardware with dedicated 16-bit tensor cores, can roughly double the raw matrix-multiply throughput — why mixed precision is close to a default setting (torch.cuda.amp, bf16=True) rather than an optional extra.

What this means in practice

Enabling mixed precision is typically a one-line change in modern frameworks and is close to free accuracy-wise, provided precision-sensitive operations (softmax, layer/RMS normalization, loss computation) are kept in fp32 internally, which automatic mixed precision handles by default. On financial data specifically, be alert to features with a wide dynamic range fed directly into a bfloat16 computation without normalization — bfloat16's reduced mantissa means two values differing only in their 8th significant digit become indistinguishable.