Model Quantization for Inference

A trained model's weights are usually stored as 32-bit floating-point numbers — plenty of precision, but expensive: every multiplication touches 32 bits of data, and every weight takes four bytes of memory and bandwidth. Quantization rounds those weights (and often the intermediate activations) down to a much coarser format, typically 8-bit integers, so the same model takes a quarter of the memory and runs several times faster on hardware that is faster at integer arithmetic than floating point. The question quantization always has to answer is how much precision you can throw away before the model's predictions actually get worse.

The analogy: measuring with a coarser ruler

Think of the difference between a ruler marked in millimetres and one marked only in centimetres. For most everyday measuring — cutting wood to roughly the right length — the centimetre ruler is perfectly fine, faster to read, and cheaper to make. But if you're measuring a part that needs to fit with a tenth-of-a-millimetre tolerance, rounding every reading to the nearest centimetre will eventually produce a part that doesn't fit. Quantizing a model's weights from 32-bit floats to 8-bit integers is exactly this: a coarser ruler, applied to every single weight, that is fine almost everywhere and occasionally costly where precision actually mattered.

Worked example 1: mapping floats to 8-bit integers

Suppose a layer's weights range from $-2.0$ to $2.0$. An 8-bit integer covers $256$ distinct values ($-128$ to $127$). The mapping (a scale factor) is:

$$\text{scale} = \frac{2.0 - (-2.0)}{256} = 0.0156$$

A weight of $0.9$ is stored as the nearest integer to $0.9 / 0.0156 = 57.7$, rounded to $58$. To recover an approximate float at inference time: $58 \times 0.0156 = 0.905$ — close to the original $0.9$, off by $0.005$. Repeated across millions of weights, these small roundings mostly cancel out, but for weights near the edges of the range the relative error is larger, which is why the scale factor is chosen per-layer (or even per-channel) to keep each layer's own range as tight as possible.

Worked example 2: the speed and size payoff

A model with 100 million parameters at 32-bit float storage takes $100{,}000{,}000 \times 4 \text{ bytes} = 400$MB. Quantized to 8-bit integers, the same model takes $100{,}000{,}000 \times 1 \text{ byte} = 100$MB — a 4x reduction, meaning 4x less data to move through memory bandwidth, which is very often the actual bottleneck in inference, not raw arithmetic. On hardware with dedicated integer instructions, the arithmetic itself can also run 2–4x faster. Combined, a quantized model commonly serves the same request in a quarter to a half the latency of its float32 original — directly shrinking the inference line-item in a latency budget.

Think of the curve above as a stand-in for a loss surface as precision drops: near the top of the range, small rounding steps barely move the loss; drag toward the steep part and the same-sized step in the input produces a much bigger jump in output — the same reason quantization error is small almost everywhere but can spike where a model's decision boundary happens to sit near a rounding edge.

What this means in practice

Teams quantize after training ("post-training quantization") for a quick, cheap win, or fold quantization into the training loop itself ("quantization-aware training") when the accuracy hit from post-training rounding is too large — the model learns weights that are already forgiving of the coarser grid it will run on. Either way, a quantized model is always validated against the same test set as the original before shipping, because the accuracy loss is model- and layer-specific, not a fixed, guaranteed-safe percentage.