The Lottery Ticket Hypothesis

Modern networks are trained with far more parameters than turn out to be necessary, after training, you can often delete 80–90% of the weights, keeping only the largest-magnitude ones, and barely lose accuracy on a re-trained version. That alone isn't surprising; it just says the network was overparameterized. What's surprising is the stronger claim: that small subnetwork, if you rewind its surviving weights back to their original random initial values, before any training happened, and train only that subnetwork alone, reaches accuracy comparable to the full network. On this view, the full network's job wasn't to learn 100% of its weights; it was to contain, somewhere inside it, a lucky combination of a good subnetwork and a good starting point for it, with enough tickets that one was bound to be a winner.

The analogy: buying every ticket to guarantee a winner

Imagine a lottery where you can't choose your numbers, you can only buy a huge stack of pre-printed tickets and see which wins. One ticket almost certainly loses. Thousands of tickets, though, means the stack is very likely to contain a winner, even though any single ticket is still just as unlikely to win alone. A large randomly-initialized network is like that stack: most individual subnetworks inside it, trained alone from their own random start, would train badly or not at all. But a network with millions of weights contains an enormous number of possible subnetworks, and the hypothesis says some ("winning tickets") happen to have both a lucky structure and a lucky starting point that make them independently trainable, standard training on the full network implicitly finds and grows one of these winners while the rest shrinks toward irrelevance.

Finding a winning ticket: iterative magnitude pruning

The hypothesis is normally tested (and winning tickets found) with a specific recipe:

1. Randomly initialize a full network, recording those initial weight values.
2. Train it normally to convergence.
3. Prune a fraction (e.g., the smallest-magnitude 20%) of the weights, permanently zeroing them out, this masks a smaller subnetwork.
4. Rewind the surviving weights to their original values from step 1, not their trained values, the step that makes it a genuine "ticket" rather than ordinary pruning.
5. Retrain only this smaller, masked subnetwork from that rewound starting point.
6. Repeat steps 3–5, pruning a bit more each round, until accuracy starts to degrade.

The surprising result: subnetworks found this way, at a fraction of the original parameter count, train to accuracy matching or nearly matching the full network, but only when rewound to their original initialization. Reinitializing randomly, or reusing trained values instead of rewinding, both perform noticeably worse.

Worked example 1: a toy pruning trace

A tiny network starts with 10 randomly initialized weights. After training, their magnitudes are: $0.9, 0.1, 0.7, 0.05, 0.8, 0.02, 0.6, 0.03, 0.75, 0.04$. Pruning the smallest 40% removes the four weights near $0.05, 0.02, 0.03, 0.04$. The surviving 6 are rewound to their own original random initial values (not the trained magnitudes shown here) and retrained from scratch. If this smaller network reaches, say, 94% accuracy versus the full network's 95%, that's consistent with a winning ticket, a 40% smaller network, from its own lucky start, doing almost the full job.

Worked example 2: why rewinding (not reinitializing) is the crux

Suppose the same 6 surviving connections instead get fresh random initial values, unrelated to what they started with, and are trained. Empirically this trains slower and lands at lower accuracy, say 88% instead of 94%. The difference isn't structure (both versions kept the same 6 connections); it's the starting point. This is the core evidence for the hypothesis: a winning ticket is a combination of sparse structure and a specific initialization, and neither alone is sufficient.

What this means in practice

Winning tickets suggest most of a large network's parameters exist to make the search for a good subnetwork likely to succeed, not because the final model needs all of them to run, the theoretical motivation behind network pruning and compression. The iterative pruning-and-rewinding procedure itself is expensive (it requires training the full network multiple times), so it's mainly a research tool for understanding why overparameterization helps, rather than a routine step in production pipelines, where cheaper pruning or distillation methods are used instead.

Practice

1. In the pruning recipe above, what specifically distinguishes "finding a winning ticket" from ordinary post-training pruning?
2. Why does re-initializing a pruned subnetwork with fresh random weights typically train worse than rewinding to its original values?
3. If a stack of 10,000 lottery tickets is very likely to contain a winner, would you expect a network with only 100 parameters to reliably contain a winning ticket at 10% sparsity? Why or why not?