BFloat16 and Numerical Precision Choices

A floating-point number is split into an exponent (how large or small the number's scale is) and a mantissa (how precisely its value is specified within that scale). Standard 16-bit float (FP16) takes 32-bit float's format and shrinks both parts, giving it a much narrower exponent range — values much larger than about 65,000 or much smaller than roughly $6\times10^{-5}$ simply cannot be represented and either overflow to infinity or underflow to zero, a real problem for the wide range of magnitudes that appear in neural network gradients and activations. Bfloat16 ("brain float 16") takes a different trade-off: it keeps the same 8-bit exponent as full 32-bit float, so it covers exactly the same enormous range of magnitudes, but shrinks the mantissa down to 7 bits instead of FP16's 10, meaning each individual number is represented less precisely within its range.

The practical consequence is that bfloat16 rarely overflows or underflows during training — sidestepping the loss-scaling workarounds FP16 needs — at the cost of more rounding error per number, which most deep learning workloads tolerate well since they're already robust to noisy gradients, but which can bite in numerically sensitive code that needs many accurate decimal digits. This is why modern training hardware and frameworks increasingly default to bfloat16 over FP16 for mixed-precision training: it removes an entire category of "why did my loss become NaN" debugging without requiring loss scaling at all, at the cost of a model that carries slightly coarser rounding throughout, which for most gradient-based training washes out over many steps rather than compounding into a real problem.