Network Pruning and Structured Sparsity

A trained network often has far more weights than it strictly needs — many contribute almost nothing to the output. If a model has to run within a strict latency budget, like a signal-generation network that must produce an output before the next tick arrives, shrinking it after training can buy real speed with only a small accuracy cost. But "shrinking" can mean two very different things, and only one of them actually makes the model run faster on ordinary hardware.

The analogy: trimming leaves versus trimming branches

Snip individual leaves off a hedge one at a time and the hedge barely changes shape, but it is no easier to wheel through a narrow gate — it's still the same overall size. Cut off whole branches instead, and you get an honestly smaller hedge. Unstructured pruning zeroes out individual weights scattered anywhere in a matrix (the leaves); structured pruning removes entire neurons, channels, or filters (the branches), leaving behind an actually smaller dense network.

The mechanics

The simplest criterion is magnitude: keep a weight $w$ if $|w| > \tau$, where the threshold $\tau$ is chosen to hit a target sparsity level (e.g., remove the smallest 80% of weights by absolute value). In words: assume that weights close to zero are contributing little, and delete the ones with the smallest magnitude first, checking accuracy doesn't collapse. Structured pruning applies the same idea one level up — scoring whole filters or neurons (e.g., by the sum of their weights' magnitudes) and removing the lowest-scoring ones entirely.

Worked example 1: unstructured pruning

A toy layer has 10 weights: $0.9, -0.1, 0.4, 0.02, 0.7, -0.6, 0.05, 0.3, -0.8, 0.15$. Sorted by absolute value, the smallest 6 are $0.02, 0.05, 0.1, 0.15, 0.3, 0.4$. Zeroing those leaves only $0.9, 0.7, -0.6, -0.8$ nonzero — 60% sparsity. The matrix now has scattered zeros anywhere, so a standard dense matrix-multiply routine still has to touch every one of the original 10 positions; nothing about the computation actually shrank.

Worked example 2: structured pruning

A toy convolution layer has 4 filters with importance scores (say, the L1 norm of each filter's weights): $3.2, 0.4, 2.1, 0.3$. Remove the lowest two ($0.4, 0.3$), keeping filters 1 and 3. The layer now genuinely has 2 filters instead of 4 — half the output channels, half the multiply-accumulate operations for this layer, and every downstream layer's input width also shrinks correspondingly. This is a real, honest speedup measurable in wall-clock time, not just a count of zeros.

What this means in practice

Pruning is usually followed by a round of fine-tuning to recover accuracy lost from removing weights, and is often combined with other compression techniques like quantization or distillation for models that must fit a strict inference-latency budget.