The Lottery Ticket Hypothesis

Modern networks are trained with far more parameters than turn out to be necessary — after training, you can often delete 80–90% of the weights, keeping only the largest-magnitude ones, and barely lose accuracy on a re-trained version. That alone isn't surprising; it just says the network was overparameterized. What's surprising is the stronger claim: that small subnetwork, if you rewind its surviving weights back to their original random initial values — before any training happened — and train only that subnetwork alone, reaches accuracy comparable to the full network. On this view, the full network's job wasn't to learn 100% of its weights; it was to contain, somewhere inside it, a lucky combination of a good subnetwork and a good starting point for it, with enough tickets that one was bound to be a winner.

The analogy: buying every ticket to guarantee a winner

Imagine a lottery where you can't choose your numbers — you can only buy a huge stack of pre-printed tickets and see which wins. One ticket almost certainly loses. Thousands of tickets, though, means the stack is very likely to contain a winner, even though any single ticket is still just as unlikely to win alone. A large randomly-initialized network is like that stack: most individual subnetworks inside it, trained alone from their own random start, would train badly or not at all. But a network with millions of weights contains an enormous number of possible subnetworks, and the hypothesis says some ("winning tickets") happen to have both a lucky structure and a lucky starting point that make them independently trainable — standard training on the full network implicitly finds and grows one of these winners while the rest shrinks toward irrelevance.

Finding a winning ticket: iterative magnitude pruning

The hypothesis is normally tested (and winning tickets found) with a specific recipe:

1. Randomly initialize a full network, recording those initial weight values.
2. Train it normally to convergence.
3. Prune a fraction (e.g., the smallest-magnitude 20%) of the weights, permanently zeroing them out — this masks a smaller subnetwork.
4. Rewind the surviving weights to their original values from step 1, not their trained values — the step that makes it a genuine "ticket" rather than ordinary pruning.
5. Retrain only this smaller, masked subnetwork from that rewound starting point.
6. Repeat steps 3–5, pruning a bit more each round, until accuracy starts to degrade.

The surprising result: subnetworks found this way, at a fraction of the original parameter count, train to accuracy matching or nearly matching the full network — but only when rewound to their original initialization. Reinitializing randomly, or reusing trained values instead of rewinding, both perform noticeably worse.

Worked example 1: a toy pruning trace

A tiny network starts with 10 randomly initialized weights. After training, their magnitudes are: $0.9, 0.1, 0.7, 0.05, 0.8, 0.02, 0.6, 0.03, 0.75, 0.04$. Pruning the smallest 40% removes the four weights near $0.05, 0.02, 0.03, 0.04$. The surviving 6 are rewound to their own original random initial values (not the trained magnitudes shown here) and retrained from scratch. If this smaller network reaches, say, 94% accuracy versus the full network's 95%, that's consistent with a winning ticket — a 40% smaller network, from its own lucky start, doing almost the full job.

Worked example 2: why rewinding (not reinitializing) is the crux

Suppose the same 6 surviving connections instead get fresh random initial values, unrelated to what they started with, and are trained. Empirically this trains slower and lands at lower accuracy — say 88% instead of 94%. The difference isn't structure (both versions kept the same 6 connections); it's the starting point. This is the core evidence for the hypothesis: a winning ticket is a combination of sparse structure and a specific initialization, and neither alone is sufficient.

What this means in practice

Winning tickets suggest most of a large network's parameters exist to make the search for a good subnetwork likely to succeed, not because the final model needs all of them to run — the theoretical motivation behind network pruning and compression. The iterative pruning-and-rewinding procedure itself is expensive (it requires training the full network multiple times), so it's mainly a research tool for understanding why overparameterization helps, rather than a routine step in production pipelines, where cheaper pruning or distillation methods are used instead.

Practice

1. In the pruning recipe above, what specifically distinguishes "finding a winning ticket" from ordinary post-training pruning?
2. Why does re-initializing a pruned subnetwork with fresh random weights typically train worse than rewinding to its original values?
3. If a stack of 10,000 lottery tickets is very likely to contain a winner, would you expect a network with only 100 parameters to reliably contain a winning ticket at 10% sparsity? Why or why not?