Dropout,
explained.

This is rung 1, so the comparison is not against another rung but against the practice it displaced. Before dropout, the standard defences against overfitting were an L2 penalty on the weights, stopping training early, and — where you could afford it — training several networks independently and averaging their predictions. The first two shrink the model's capacity. The third works far better and costs you a full training run per member.

Dropout's change is to get the averaging without the extra runs. On every forward pass, each unit is switched off independently with probability p, so each pass trains a different thinned network, and the weights are shared between all of them. With n droppable units there are 2^ⁿ such networks, and the single set of weights you end up with approximates averaging over that whole collection.

The units to carry up the ladder: dropout introduces exactly one hyperparameter, the rate p (classically 0.5 for hidden units, lower for inputs), and it adds no parameters at all. It also introduces the first train/test asymmetry on this ladder — the network you train is never the network you deploy — and the bookkeeping that fixes it, scaling the survivors by 1/(1−p) so the expected input to the next layer is unchanged. Rung 2 introduces a second, more awkward asymmetry of the same kind, and rung 5 removes it.

A big network has enough capacity to memorize its training set — to fit noise instead of signal. One face of this is co-adaptation: neurons learn to work only in specific committees, each one correcting for another's quirks. A feature detector that only fires usefully when three particular neighbors are present is fragile; it hasn't learned something true about the data, it has learned to fit this data alongside those partners. Come test time, on data it hasn't seen, the brittle committee falls apart.

The pre-dropout cure was to train many separate networks and average them — a real ensemble, which reliably generalizes better. But training and running dozens of large networks is expensive. Dropout's genius is getting the ensemble benefit from a single network, almost for free.

On each training step, for each unit in a dropout layer, flip a biased coin. With probability p (say 0.5) set that unit's output to zero for this step — as if it doesn't exist. The remaining units carry the forward pass and receive the backward pass; the dropped ones are simply absent, updated on other steps when they survive.

Because a neuron never knows which of its inputs will be present, it cannot lean on any single partner. It is pushed to extract a feature that is useful on average, across all the random committees it finds itself in. That is the regularization, mechanically.

Zeroing a fraction p of units shrinks a layer's total output by roughly that fraction. If training runs at that reduced scale but testing runs the full network, the next layer sees a different input magnitude in the two regimes — and everything downstream is thrown off. Inverted dropout fixes this at training time by dividing the surviving activations by (1 − p), which restores the expected output exactly:

The mask's mean is (1 − p), so dividing by (1 − p) makes the expected surviving activation equal the original — the layer is unbiased. That single line is why test time needs no dropout and no rescaling: the network was already trained to produce, in expectation, exactly what the full network produces. The runnable version below confirms the expectation matches the plain forward pass to machine precision.

Here is the deeper reason it works. Every distinct dropout mask picks out a different sub-network — a subset of neurons wired through the same shared weights. A layer of n units has 2ⁿ possible masks, so over training you are sampling from, and improving, an exponentially large family of thinned networks that all share parameters. Training with dropout approximately trains this whole ensemble at once.

At test time, running the full network with the scaled weights is a cheap approximation to averaging that entire ensemble's predictions. And averaging many models is the oldest trick in the book for cutting variance and beating overfitting — you are getting the statistical benefit of thousands of models for the price of one. With just 100 units, that is 2¹⁰⁰ ≈ 10³⁰ sub-networks, more than there are stars in the observable universe.

The JMLR paper writes p for the probability of keeping a unit, the reverse of the drop rate that most frameworks take and that the rest of this page uses. Its p = 0.5 means keep half; its typical input-layer value, p = 0.8, drops one input in five.

1. Sample a mask for every training case. For each unit j in layer l, draw r_j ~ Bernoulli(p) and multiply it into that unit's output, so the next layer receives r ∗ y. Each example in a mini-batch gets its own mask, so one mini-batch trains many different thinned networks.
2. Backpropagate through the thinned network only. Gradients flow through the surviving units. A parameter that a training case did not use gets a zero gradient from that case, and gradients are averaged over the mini-batch as usual.
3. Cap each unit's incoming weights. After every update, if the vector of weights entering a hidden unit has an L2 norm above a constant c, it is projected back onto the sphere of radius c. The paper's practical guide puts c at 3 to 4, and this max-norm constraint is the regularizer the authors found most useful alongside dropout.
4. Drive the optimizer hard. Because the noisy gradients partly cancel, the guide recommends 10 to 100 times the learning rate that suits the same network without dropout, momentum of 0.95 to 0.99 instead of the usual 0.9, and a decaying schedule. Max-norm is what stops those large steps from inflating the weights.
5. Size the layer for the units that survive. Only about pn of n units are active on a given case, so if n units suit a standard network, the guide suggests at least n/p with dropout.
6. Test with one scaled network. At test time nothing is dropped. Instead, the weights leaving each unit are multiplied by its keep probability, W_test = pW, so each unit's input equals its expected input during training. The paper also notes the equivalent alternative of multiplying retained activations by 1/p during training and leaving test weights alone, which is the form frameworks implement.

How much noise, and of what kind

Typical keep probabilities are 0.5 to 0.8 for hidden units and 0.8 for real-valued inputs such as image patches or speech frames; on MNIST, test error was flat for p between 0.4 and 0.8. Bernoulli masks are not the only choice. The paper reports that multiplying each activation by Gaussian noise with mean 1 and variance (1 − p)/p, the same variance as a 1/p-scaled Bernoulli mask, works as well or slightly better, and because its mean is 1 it needs no rescaling at test time.

Where the recipe was run

The same procedure was applied to fully connected networks on MNIST; to convolutional networks on SVHN, CIFAR and ImageNet, on SVHN with keep rates rising from 0.5 in the top layers to 0.9 at the input; to deep networks for TIMIT speech, with and without deep-belief-net pretraining; to bag-of-words classifiers on Reuters; and to a small genetics data set where it was set against Bayesian neural networks. The paper also extends dropout to restricted Boltzmann machines, and it checks the cost of the test-time shortcut: averaging the predictions of k randomly thinned networks only matched the weight-scaled network at about k = 50.

In the paper's notation, with p the keep probability, the forward pass into layer l + 1 and the test-time rule are:

For linear regression with dropped-out inputs, the noise can be averaged out exactly, which shows what dropout penalizes:

• R — an N × D matrix of independent Bernoulli(p) entries, one per input value; ∗ is the elementwise product.
• ‖y − pXw‖² — the ordinary squared error of the scaled-weight model that is used at test time.
• p(1 − p)‖Γw‖² — a ridge penalty that scales the cost of each weight w_d by the spread of feature d across the data (Γ holds the square root of each feature's sum of squares), so weights on features that vary a lot are squeezed hardest.

At the extremes, p = 1 removes the penalty and gives plain least squares, while p(1 − p) peaks at 0.25 when p = 0.5. Absorbing p into the weights, w̃ = pw, rewrites the penalty as ((1 − p)/p)‖Γw̃‖², which grows without limit as p → 0: very aggressive dropout behaves like very strong shrinkage.

The paper runs one recipe across vision, speech, text and genetics and compares each result with networks trained without dropout or with the best published result at the time. Error rates, lower is better:

Two comparisons isolate the ingredients. On one MNIST network (784-1024-1024-2048-10, ReLU) the authors swapped regularizers: L2 alone gave 1.62%, L2 with KL-sparsity 1.55%, max-norm alone 1.35%, dropout with L2 1.25% and dropout with max-norm 1.05%. On SVHN, adding dropout to the convolutional layers as well as the fully connected ones lowered error further, which the authors attribute to the noisier inputs it feeds the upper layers.

The negative results are reported too. On text the gain was much smaller than on images or speech. On MNIST subsets of 100 and 500 examples dropout gave no improvement, because a large network can still memorize so little data; the benefit grew with data and then shrank once the set was too large to overfit. On the small alternative-splicing data set a Bayesian neural network beat it, 623 to 567 bits of code quality (higher is better), although dropout still beat early stopping at 440 and was far cheaper to train and run.

The paper is direct about the cost: the gradient each pass produces is noisier, because it is a gradient for one thinned network rather than the full one, so training takes substantially longer to converge — the authors put it at roughly two to three times the number of epochs for the same result. Dropout buys generalisation with compute, which is a trade worth naming because every later regulariser on this ladder is measured against it.

They are also clear that it is not free at inference time to reason about: the weight-scaling rule is an approximation to averaging over the 2^ⁿ networks, only a rough stand-in for that average in a deep nonlinear network, and the paper presents it as an approximation rather than claiming the ensemble interpretation is literal.

How it held up: unevenly, and the unevenness is instructive. Dropout is still standard inside transformer blocks and on classifier heads, and the idea — inject noise into the computation to stop units from co-adapting — spawned a family (DropConnect, DropPath, stochastic depth). But in convolutional networks it was largely displaced by rung 2, which regularises as a side effect of what it is actually for, and in large language-model pre-training it is frequently set to zero: when a model makes roughly one pass over a corpus far larger than its capacity, the overfitting dropout exists to prevent is not the binding constraint. A regulariser whose value depends on the data-to-parameter ratio is one you should expect to see switched off as that ratio moves, which is the axis the scaling ladder formalises.

Dropout was, for a stretch, the reason deep networks generalized — a central ingredient in the models that kicked off the deep-learning boom. Its influence outlived its dominance. The ensemble-by-noise idea seeded a whole family: DropConnect (drop weights, not units), stochastic depth (drop whole layers), DropPath, and the general principle that injecting the right noise during training is a regularizer.

In today's largest models dropout is often lighter or absent — batch and layer normalization, weight decay, data augmentation, and simply training on enormous datasets carry much of the anti-overfitting load. But open a transformer implementation and you will still find small dropout on the attention weights and feed-forward layers. The specific tool receded; the idea it proved — that a single network can behave like an ensemble — is permanent.

Dropout was a beautifully simple answer to overfitting: during training, randomly "drop" a fraction of neurons on each forward pass, forcing the network to not rely on any single unit. Because the network never sees the same architecture twice, it can't build fragile co-adaptations where neurons only work in specific combinations — so it learns more robust, redundant features that generalize better.

One way to see it: dropout approximates training an enormous ensemble of thinned networks that share weights, then averaging them at test time (when dropout is off and activations are scaled to compensate). Ensembling is a classic way to reduce overfitting; dropout gets much of that benefit for almost no extra cost.

That train-vs-test difference is the practical gotcha it made everyone learn: dropout is active only during training, and inference must either scale activations or use the "inverted dropout" trick so the expected output matches — a bug that silently hurts models when handled wrong.

Dropout remains a standard regularizer, though its role shifted as the field changed: heavy data augmentation, batch/layer normalization, and simply training on more data reduced reliance on it in some settings, and large Transformers often use only light dropout. But the core idea — inject noise during training to prevent brittle co-adaptation — endures, and variants (spatial dropout, DropPath, attention dropout) are everywhere in modern architectures.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →