Winning
tickets.

Modern networks are wildly over-parameterized — millions or billions of weights for tasks that seemingly need far fewer. We've long known you can prune a trained network, deleting most of its weights, and keep almost all the accuracy. The puzzle was always: if the small network is enough, why couldn't we just train the small network directly? In practice, training a comparably small network from scratch gave worse results. Over-parameterization seemed necessary for training, even if not for the final function.

The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) offered a startling answer. A dense, randomly-initialized network contains a sparse subnetwork — a winning ticket — that can be trained in isolation to full accuracy, provided you start it from the same initial weights it had inside the big network. The dense network is a bag of lottery tickets: many candidate subnetworks, each with its own random initialization, and training reveals which ones held winning numbers. Over-parameterization helps not (only) because big functions are better, but because more tickets means better odds that some subnetwork got a lucky initialization.

The recipe is iterative magnitude pruning, and it's remarkably simple:

The pruning criterion is just magnitude: after training, weights that ended up near zero are judged least important and cut, leaving a binary mask of kept (1) and removed (0) connections. Doing it iteratively — prune a modest fraction, retrain, prune again — finds much sparser winning tickets than pruning all at once, because each round re-evaluates importance on the smaller network. After k rounds at pruning fraction p, only (1−p)^k of the weights remain; a few rounds at 20% gets you to a small fraction of the original. What survives is a sparse skeleton — but the skeleton alone isn't the ticket.

Here is the finding that made the paper famous. Take the winning ticket — the sparse mask you found — and consider three ways to initialize its surviving weights before training it in isolation:

The mask alone — a good sparse architecture — is not enough. What makes a ticket win is the specific combination of which connections survive and the exact random values those connections started with. Randomly reinitializing the same structure throws away the winning numbers. That's why it's a lottery: the surviving weights happened, by chance, to be initialized in a configuration that gradient descent could ride to a good solution. The reset-to-init step preserves that lucky draw. It's a claim about the entanglement of structure and initialization — not just that small networks can work, but that they were secretly trainable all along, if you knew the ticket.

A winning ticket is the original network f(x; θ₀) restricted by a binary mask m, trained from that same θ₀:

θ₀ are the original init weights; m is the keep/drop mask from magnitude pruning (⊙ is elementwise product). The ticket trains from m ⊙ θ₀ — masked structure, original initialization.

Two quantities describe the ticket. Sparsity is the fraction of weights removed (the zeros in m); remaining density is 1 − sparsity. Under iterative pruning, if you remove a fraction p each round, the fraction of weights still alive after k rounds is multiplicative:

Pruning compounds: each round cuts a fraction of what's left, so density decays geometrically. The runnable version below builds a magnitude mask, applies it, measures sparsity, resets survivors to init, and computes the compounding.

The experiments were clean and the claims precise:

The caveats matter. Finding tickets by full iterative pruning is expensive — you train the dense network many times over — so it doesn't (yet) save training compute; it explains something instead. And at very large scale the original "reset to step 0" recipe is finicky; follow-up work found you often must reset to the weights after a few hundred early training steps ("rewinding") rather than to the pristine init. But the core empirical result held up and reoriented a whole line of research.

The Lottery Ticket Hypothesis launched a wave of work on sparse training: if trainable sparse subnetworks exist from the beginning, can we find them cheaply — or even at initialization, before training — and then train sparse from the start to save compute and memory? Methods like SNIP, GraSP, and the "Rigging the Lottery" (RigL) line chase exactly this, trying to skip the expensive prune-retrain loop. Others study whether tickets transfer across tasks and datasets, which would make a found ticket a reusable asset.

It also connects to the broader efficiency story. Where knowledge distillation compresses a model by retraining a small student, and low-rank adaptation makes fine-tuning cheap by constraining updates, the lottery ticket line attacks the same goal — capable models at a fraction of the cost — from the angle of sparse structure. And it deepened a genuine mystery about why deep, over-parameterized networks are so trainable in the first place. Even where the practical payoff is still being worked out, few papers reshaped how the field pictures the inside of a neural network as sharply as this one: not a monolith that needs all its weights, but a haystack full of winning tickets.