Quantisation and Pruning for Latency

A model that scores well in research can still be unusable in production if it's too slow to run within a trading system's latency budget — if a decision needs to be made in microseconds and the model takes milliseconds, the model is simply the wrong tool no matter how accurate it is. Rather than accepting that trade-off or redesigning the model from scratch, two standard techniques shrink a trained model's computational cost directly: quantization and pruning.

Two ways to shrink a model

Quantization reduces the numerical precision used to store and compute with a model's weights — typically from 32-bit floating-point down to 8-bit integers. Lower-precision arithmetic is faster and the model file is smaller, at the cost of precision: an 8-bit integer represents only 256 distinct values versus billions for a 32-bit float, so each weight becomes a coarser approximation of its trained value.

Pruning removes parts of a model that contribute little to its output — weights near zero, or entire neurons whose removal barely changes validation predictions — shrinking the model's actual size and computation, not just each computation's precision. A fully trained network is often over-parameterized: many connections contribute almost nothing, and removing them costs little accuracy while directly cutting compute.

Both trade a small, controlled amount of accuracy for a measurable speed gain, and both should be validated against held-out data before deployment, never assumed safe by default.

Worked example: quantifying the trade-off

Suppose a model in 32-bit floating point scores 91.0% validation accuracy and takes 4.0 milliseconds per prediction — too slow for a 1.0-millisecond latency budget. Quantizing to 8-bit integers cuts inference time roughly fourfold, to about 1.0 millisecond, right at budget, while accuracy drops slightly to 90.6% — a small, acceptable cost for now fitting the requirement.

If the real budget is 0.5 milliseconds, pruning can be applied on top: removing the 30% of weights with smallest magnitude brings inference down to roughly 0.6 milliseconds, with accuracy at 90.1%. Combined, the model goes from 4.0ms/91.0% to 0.6ms/90.1% — an 85% latency reduction for under one percentage point of accuracy, often clearly worth it once the alternative is "unusable in production."

Treat that curve as "accuracy retained" against "amount of pruning applied": it stays nearly flat while removing the least useful weights, then drops off sharply once pruning starts cutting into weights the model actually needs — which is exactly why pruning has to be validated at each step rather than applied all at once and checked only at the end.

What this means in practice

Neither technique is free, and both need validating like any other model change — on held-out data, since the point where accuracy starts degrading is specific to each model. Teams typically apply quantization first (the larger, cheaper win) and add pruning only if latency requirements still aren't met, checking accuracy at each step rather than stacking both blindly.