Pruning and Model Compression

A trained neural network is typically full of redundancy: many weights are near zero and contribute almost nothing to the output, or entire neurons rarely activate strongly enough to matter. Pruning removes that dead weight — literally — shrinking the model so it runs faster and uses less memory at inference time, ideally with a negligible drop in accuracy. It's one of the standard tools for fitting a model into a tight latency budget once training is done and accuracy is already acceptable.

Two flavors of pruning

Unstructured pruning zeroes out individual weights below some magnitude threshold, wherever they sit in the network. It can remove a large fraction of parameters — 80% or more in many networks — with little accuracy loss, but the result is a "sparse" matrix with scattered zeros, and ordinary hardware doesn't automatically run sparse matrix multiplication faster unless paired with specialized sparse-computation libraries. Structured pruning removes whole neurons, channels, or attention heads at once, keeping the remaining network dense and regularly shaped. This yields a smaller but ordinary matrix that runs faster on standard hardware immediately, at the cost of usually being able to remove a smaller fraction of parameters for the same accuracy hit, since removing a whole neuron is a coarser cut than removing individual weights.

The typical workflow is iterative: prune a chunk of the smallest-magnitude weights or lowest-importance neurons, briefly retrain ("fine-tune") the remaining network to recover accuracy, then repeat, rather than pruning everything in one aggressive pass.

Worked example

A latency-sensitive model scoring order-book snapshots has 4 million parameters and takes 800 microseconds per inference — too slow for a 500-microsecond budget. The team applies structured pruning, removing the 30% of neurons with the smallest average activation magnitude across a validation set, then fine-tunes for two epochs to recover accuracy. The pruned model has 2.8 million parameters, runs in 540 microseconds, and its validation accuracy drops from 91.2% to 90.6% — a small cost for meeting most of the target. A second pruning pass, removing another 15% of neurons, gets latency down to 410 microseconds but accuracy falls to 87.9%, judged too steep a drop; the team keeps the first pass's result and looks at quantization instead to close the remaining gap.

What this means in practice

Pruning decisions should be made against the actual accuracy metric a strategy cares about, not just a generic validation loss — a model can lose most of its accuracy on rare, high-impact tail events while barely moving on the average-case metric, which is exactly the failure mode a trading desk cannot afford. Always re-evaluate a pruned model on the same held-out set and, ideally, the same downstream P&L-style backtest used to validate the original model before deploying it.