Winning
tickets.

Rungs 1 through 6 are all answers to "how do we train this network successfully". This rung asks what the successful training actually needed, and it is the only one here that is a claim about networks rather than a technique for building them. Pruning a trained network down to a small fraction of its weights was already known to preserve accuracy. The open question was whether that small subnetwork could have been trained on its own, or whether the discarded weights were load-bearing during training.

The finding is that it could — with one condition that is the whole result. Take the pruned subnetwork's connections, and reset its surviving weights to the exact values they were initialised with, not to fresh random ones. Trained from that initialisation, in isolation, it matches the full network's accuracy in a comparable number of steps. Reinitialise the same structure randomly and it does not. So what the procedure finds is not a good architecture; it is a structure paired with a particular starting point — the winning ticket.

Stated in units: the surviving fraction is reported in the region of 10–20% of the original weights on the small image classifiers used, found by iterative magnitude pruning — train, remove the smallest-magnitude fraction, rewind the rest, repeat. Which also states the cost honestly: the method needs many full training runs to find a network you could have trained in one, so as stated it is an existence proof, not a saving.

Modern networks are wildly over-parameterized — millions or billions of weights for tasks that seemingly need far fewer. We've long known you can prune a trained network, deleting most of its weights, and keep almost all the accuracy. The puzzle was always: if the small network is enough, why couldn't we just train the small network directly? In practice, training a comparably small network from scratch gave worse results. Over-parameterization seemed necessary for training, even if not for the final function.

The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) offered a startling answer. A dense, randomly-initialized network contains a sparse subnetwork — a winning ticket — that can be trained in isolation to full accuracy, provided you start it from the same initial weights it had inside the big network. The dense network is a bag of lottery tickets: many candidate subnetworks, each with its own random initialization, and training reveals which ones held winning numbers. Over-parameterization helps not (only) because big functions are better, but because more tickets means better odds that some subnetwork got a lucky initialization.

The recipe is iterative magnitude pruning, and it's remarkably simple:

The pruning criterion is just magnitude: after training, weights that ended up near zero are judged least important and cut, leaving a binary mask of kept (1) and removed (0) connections. Doing it iteratively — prune a modest fraction, retrain, prune again — finds much sparser winning tickets than pruning all at once, because each round re-evaluates importance on the smaller network. After k rounds at pruning fraction p, only (1−p)^^k of the weights remain; a few rounds at 20% gets you to a small fraction of the original. What survives is a sparse skeleton — but the skeleton alone isn't the ticket.

Here is the finding that made the paper famous. Take the winning ticket — the sparse mask you found — and consider three ways to initialize its surviving weights before training it in isolation:

The mask alone — a good sparse architecture — is not enough. What makes a ticket win is the specific combination of which connections survive and the exact random values those connections started with. Randomly reinitializing the same structure throws away the winning numbers. That's why it's a lottery: the surviving weights happened, by chance, to be initialized in a configuration that gradient descent could ride to a good solution. The reset-to-init step preserves that lucky draw. It's a claim about the entanglement of structure and initialization — not just that small networks can work, but that they were secretly trainable all along, if you knew the ticket.

A winning ticket is the original network f(x; θ₀) restricted by a binary mask m, trained from that same θ₀:

θ₀ are the original init weights; m is the keep/drop mask from magnitude pruning (⊙ is elementwise product). The ticket trains from m ⊙ θ₀ — masked structure, original initialization.

Two quantities describe the ticket. Sparsity is the fraction of weights removed (the zeros in m); remaining density is 1 − sparsity. Under iterative pruning, if you remove a fraction p each round, the fraction of weights still alive after k rounds is multiplicative:

Pruning compounds: each round cuts a fraction of what's left, so density decays geometrically. The runnable version below builds a magnitude mask, applies it, measures sparsity, resets survivors to init, and computes the compounding.

1. Initialize and save. Draw θ₀ from the network's normal initialization (Gaussian Glorot in these experiments) and keep a copy.
2. Train to completion. Train for j iterations on the usual schedule, giving θ_j.
3. Prune by magnitude, layer by layer. The criterion is the one earlier pruning work, such as the 2015 method of Han and colleagues, used to compress trained networks: remove the lowest-magnitude fraction of each layer's weights, 20% per round for Lenet's fully connected layers, 10% for the convolutional layers of Conv-2 and Conv-4 and 15% for Conv-6, and half the usual rate for connections into the output layer.
4. Rewind the survivors to θ₀. The candidate is f(x; m ⊙ θ₀): the mask comes from the trained weights, the values from initialization.
5. Repeat. Retrain the masked network, prune again from its new trained weights, rewind again. After n rounds at rate r a layer keeps (1 − r)^ⁿ of its weights; Lenet's reported sizes of 51.3%, 21.1% and 3.6% correspond to about 3, 7 and 15 rounds at 20%.
6. Run the controls. Train each mask again from a fresh random initialization, and compare one-shot pruning, a single cut to the target size, with the iterative schedule. Reported curves average five trials.
7. Judge by two numbers. The iteration of minimum validation loss (the early-stopping point, used as the measure of learning speed) and the test accuracy at that iteration. A winning ticket must be no slower and no less accurate than the dense network.

The networks

Lenet-300-100 (266,000 weights) on MNIST, trained for 50,000 iterations at batch 60 with Adam at 1.2 × 10^⁻³. Conv-2, Conv-4 and Conv-6 on CIFAR-10: scaled-down VGG-style networks with two, four or six 3 × 3 convolutional layers, max pooling after every pair, and fully connected layers of 256, 256 and 10 units, trained with Adam at 2–3 × 10^⁻⁴ for 20,000 to 30,000 iterations. Then two deeper CIFAR-10 networks, VGG-19 (20 million weights) and a ResNet-18 of the small CIFAR design, three stages of 16, 32 and 64 filters with about 270,000 weights rather than the ImageNet model, trained with SGD and momentum 0.9 at a learning rate of 0.1 stepped down tenfold twice. For these two the authors switch to global pruning, a single magnitude threshold across all convolutional layers at 20% per round, because, the authors conjecture, pruning each layer at the same rate made small layers bottlenecks: VGG-19's first two layers have 1,728 and 36,864 weights, its last 2.35 million.

The learning-rate condition

At the learning rate that trains VGG-19 and ResNet-18 best, 0.1, iterative pruning found no winning tickets: rewound subnetworks did no better than random reinitialization. At 0.01 the usual pattern returned, but those subnetworks fell short of what the dense network reaches at 0.1; ResNet-18's best reached 89.5% against 90.5%. Linear learning-rate warmup restored the effect: 10,000 warmup iterations at a rate of 0.1 for VGG-19, and 20,000 at 0.03 for ResNet-18.

• Dθ — the initialization distribution the network normally uses.
• m — the binary mask; ‖m‖₀ counts the weights kept.
• j′ ≤ j — commensurate training time, measured at the minimum of validation loss.
• a′ ≥ a — commensurate accuracy, on the test set at that point.
• θ₀′ — the control: same mask, new values; the claim is that this one fails.

Extremes: m = all ones satisfies the statement trivially, since the dense network is its own ticket, so all of the content is in ‖m‖₀ ≪ |θ|. Push sparsity far enough and a′ ≥ a must fail; for Lenet it held down to about 3.6% of the weights, where tickets fell back to the dense network's accuracy. The statement is existential: it says such an m exists, and iterative magnitude pruning is only the paper's way of finding one, so a failure of that procedure on a larger network does not by itself refute it.

The controls carry the argument. The same masks with fresh random initializations learned more slowly than the dense network and lost accuracy after little pruning, so structure alone was not enough. One-shot pruning also found tickets, reaching minimum validation loss sooner than the dense Lenet between 67.5% and 17.6% of weights, but iterative pruning found ones that learned faster and stayed accurate at smaller sizes. Tickets survived dropout (rate 0.5) and the gains stacked: on top of dropout's own improvement, pruning added up to 2.3, 4.6 and 4.7 points for Conv-2, Conv-4 and Conv-6. Tickets also generalized better: like the dense networks they reached 100% training accuracy, but with a smaller gap to test accuracy.

The negative result is the deeper networks' dependence on warmup: without it, at their standard learning rate, iterative pruning found nothing better than random reinitialization. That condition is where later work, which rewinds to weights from early in training instead of θ₀, picked up.

The paper is explicit about its evidence base — small networks on small image datasets — and about the expense of iterative magnitude pruning. It frames the result as a hypothesis, which is the correct register for a claim of this shape, and does not assert that winning tickets exist in arbitrarily large models.

That caution was warranted. Scaling the procedure to large networks needed a modification that weakens the headline: rewinding to the original initialisation stops working, and you have to rewind to the weights from some early training step instead. Whether the specific initialisation matters, or whether what is really being found is a trainable sparse structure plus any sufficiently early state, is contested — work by Liu and colleagues and by Gale and colleagues found that under fair training budgets, randomly-initialised sparse networks and one-shot magnitude pruning are stronger baselines than the original comparison implied.

How it held up: as a research programme it has been enormously productive — it made "what does initialisation determine" a measurable question. As an engineering technique it mostly has not, and the reason is hardware rather than machine learning: unstructured sparsity scattered across a weight matrix gives little wall-clock speed-up on hardware built for dense matrix multiplication, so the compression that actually ships is structured — distillation to a smaller dense model, quantisation, or the sparsity-by-routing that skips whole subnetworks per token instead of individual weights.

The Lottery Ticket Hypothesis launched a wave of work on sparse training: if trainable sparse subnetworks exist from the beginning, can we find them cheaply — or even at initialization, before training — and then train sparse from the start to save compute and memory? Methods like SNIP, GraSP, and the "Rigging the Lottery" (RigL) line chase exactly this, trying to skip the expensive prune-retrain loop. Others study whether tickets transfer across tasks and datasets, which would make a found ticket a reusable asset.

It also connects to the broader efficiency story. Where knowledge distillation compresses a model by retraining a small student, and low-rank adaptation makes fine-tuning cheap by constraining updates, the lottery ticket line attacks the same goal — capable models at a fraction of the cost — from the angle of sparse structure. And it deepened a genuine mystery about why deep, over-parameterized networks are so trainable in the first place. Even where the practical payoff is still being worked out, few papers reshaped how the field pictures the inside of a neural network as sharply as this one: not a monolith that needs all its weights, but a haystack full of winning tickets.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →