What is dropout in a neural network?

Dropout is a regularization technique (Srivastava, Hinton et al., 2014) that randomly sets a fraction p of a layer's units to zero on each training step. This stops units from co-adapting — relying on specific other units being present — so each learns a more robust, independently useful feature. It is one of the most effective and widely used ways to reduce overfitting in deep networks.

Why does dropout scale the surviving activations?

Because dropping a fraction p of units lowers the layer's total output, which would shift the scale seen by the next layer between training and test time. Inverted dropout divides the surviving activations by (1 - p) during training, which makes the expected output equal to the undropped output. That way the network runs at full capacity at test time with no dropout and no rescaling needed.

Why is dropout like an ensemble?

Each random mask defines a different sub-network that shares weights with all the others. A layer of n units has 2^n possible masks, so training with dropout is approximately training an exponentially large ensemble of thinned networks at once. At test time, using all units with scaled weights approximates averaging that whole ensemble — and ensembling is a classic way to reduce variance and overfitting.

What dropout rate should I use, and where?

A common default is p = 0.5 for fully connected hidden layers and a smaller rate (0.1–0.2) for inputs. Dropout is applied only during training, never at inference. In modern architectures it is often lighter or replaced: batch and layer normalization provide some regularization, and transformers typically use small dropout on attention and feed-forward layers.

Paper Breakdowns / Dropout

Paper 45~11 min readJMLR 2014worked math + runnable code

Paper Breakdown

Dropout,
explained.

The idea sounds like sabotage: while training a neural network, randomly switch off half its neurons on every single step. Yet this one trick became the standard cure for overfitting for years. The reason is beautiful — deleting neurons at random forces every one to be independently useful, and quietly trains an ensemble of billions of networks at once. Here's exactly how, and why the surviving neurons get scaled up.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

The co-adaptation problem The trick: drop units at random Why survivors scale up An ensemble of 2ⁿ networks Run it yourself In practice What it left behind

The co-adaptation problem

A big network has enough capacity to memorize its training set — to fit noise instead of signal. One face of this is co-adaptation: neurons learn to work only in specific committees, each one correcting for another's quirks. A feature detector that only fires usefully when three particular neighbors are present is fragile; it hasn't learned something true about the data, it has learned to fit this data alongside those partners. Come test time, on data it hasn't seen, the brittle committee falls apart.

The pre-dropout cure was to train many separate networks and average them — a real ensemble, which reliably generalizes better. But training and running dozens of large networks is expensive. Dropout's genius is getting the ensemble benefit from a single network, almost for free.

The one-sentence version

Randomly deleting neurons during training stops any neuron from relying on any other, so each must learn a feature that stands on its own — and the randomness secretly averages over an army of sub-networks.

The trick: drop units at random

On each training step, for each unit in a dropout layer, flip a biased coin. With probability p (say 0.5) set that unit's output to zero for this step — as if it doesn't exist. The remaining units carry the forward pass and receive the backward pass; the dropped ones are simply absent, updated on other steps when they survive.

What a dropout layer does per step

Sample maskEach unit kept with prob 1 − p, zeroed with prob p. A fresh mask every step.

ForwardOnly surviving units pass signal — a different thinned network each step.

ScaleDivide survivors by (1 − p) so the layer's expected output is unchanged.

Test timeNo dropout, no scaling — the full network runs at once.

Because a neuron never knows which of its inputs will be present, it cannot lean on any single partner. It is pushed to extract a feature that is useful on average, across all the random committees it finds itself in. That is the regularization, mechanically.

Why the survivors scale up

Zeroing a fraction p of units shrinks a layer's total output by roughly that fraction. If training runs at that reduced scale but testing runs the full network, the next layer sees a different input magnitude in the two regimes — and everything downstream is thrown off. Inverted dropout fixes this at training time by dividing the surviving activations by (1 − p), which restores the expected output exactly:

train: ỹ_i = m_i / (1 − p) · x_i , m_i ~ Bernoulli(1 − p)
𝔼[ỹ_i] = ( 𝔼[m_i] / (1 − p) ) · x_i = ( (1 − p) / (1 − p) ) · x_i = x_i

The mask's mean is (1 − p), so dividing by (1 − p) makes the expected surviving activation equal the original — the layer is unbiased. That single line is why test time needs no dropout and no rescaling: the network was already trained to produce, in expectation, exactly what the full network produces. The runnable version below confirms the expectation matches the plain forward pass to machine precision.

An ensemble of 2ⁿ networks

Here is the deeper reason it works. Every distinct dropout mask picks out a different sub-network — a subset of neurons wired through the same shared weights. A layer of n units has 2ⁿ possible masks, so over training you are sampling from, and improving, an exponentially large family of thinned networks that all share parameters. Training with dropout approximately trains this whole ensemble at once.

At test time, running the full network with the scaled weights is a cheap approximation to averaging that entire ensemble's predictions. And averaging many models is the oldest trick in the book for cutting variance and beating overfitting — you are getting the statistical benefit of thousands of models for the price of one. With just 100 units, that is 2¹⁰⁰ ≈ 10³⁰ sub-networks, more than there are stars in the observable universe.

RUN IT YOURSELF

Dropout, and the expectation it preserves

Inverted dropout is three lines, and its correctness is checkable exactly. This zeroes units by a mask and scales the survivors by 1/(1−p); then it enumerates all 2ⁿ masks and averages, showing the expected training output equals the plain forward pass to machine precision — which is precisely why inference uses the full network untouched. It also counts the sub-networks a mask defines. Change the drop rate p or the input and watch the expectation stay pinned to x.

CPython · WebAssembly

from itertools import product

# TRAIN: zero each unit with prob p, then scale survivors by 1/(1-p) so the
# layer's EXPECTED output is unchanged. That's why TEST time runs the full
# network with no dropout and no rescaling -- the expectation already matches.

def train_forward(x, mask, p):            # drop (mask=0), then scale survivors
    keep = 1 - p
    return [xi * mi / keep for xi, mi in zip(x, mask)]

def test_forward(x):                       # inference: full network, no dropout
    return list(x)

def expected_train_output(x, p):           # exact E over all 2^n masks
    n = len(x)
    total = [0.0] * n
    for mask in product([0, 1], repeat=n):
        prob = 1.0
        for mi in mask:
            prob *= (1 - p) if mi else p
        out = train_forward(x, mask, p)
        for i in range(n):
            total[i] += prob * out[i]
    return total

def subnet_count(n):
    return 2 ** n

x, p = [2.0, 4.0, 6.0], 0.5
print(f"one random mask [1,0,1] -> {train_forward(x, [1, 0, 1], p)}   (survivors doubled, dropped = 0)")
print(f"E[train output]        -> {expected_train_output(x, p)}")
print(f"test-time forward      -> {test_forward(x)}   (identical -- no rescaling at inference)\n")
print(f"sub-networks, {len(x)}-unit layer : {subnet_count(len(x))}")
print(f"sub-networks, 100-unit layer  : 2^100 = {subnet_count(100):.3e}")

In practice

Dropout is a training-only layer with a single knob, the rate p. The paper's guidance still holds as a starting point:

Where	Typical rate	Why
Hidden fully-connected layers	p ≈ 0.5	Maximum ensembling; these layers overfit most.
Input layer	p ≈ 0.1–0.2	Dropping raw inputs is destructive; keep it gentle.
Convolutional layers	low / none	Weight sharing already regularizes; spatial dropout if any.
Inference	0 (off)	Full network, weights already scaled — no dropout ever at test.

Two footguns worth naming. Forgetting to switch dropout off at evaluation (in frameworks, leaving the model in "train" mode) silently corrupts your metrics. And stacking heavy dropout on top of strong normalization can under-fit — modern nets often use lighter rates because other regularizers are already doing work.

What it left behind

Dropout was, for a stretch, the reason deep networks generalized — a central ingredient in the models that kicked off the deep-learning boom. Its influence outlived its dominance. The ensemble-by-noise idea seeded a whole family: DropConnect (drop weights, not units), stochastic depth (drop whole layers), DropPath, and the general principle that injecting the right noise during training is a regularizer.

In today's largest models dropout is often lighter or absent — batch and layer normalization, weight decay, data augmentation, and simply training on enormous datasets carry much of the anti-overfitting load. But open a transformer implementation and you will still find small dropout on the attention weights and feed-forward layers. The specific tool receded; the idea it proved — that a single network can behave like an ensemble — is permanent.

Worth knowing

Keep dropout on at inference and run the network many times, and the spread of predictions estimates the model's uncertainty — "Monte Carlo dropout," a cheap trick for asking a network how sure it is.

Frequently asked

Quick answers

What does dropout actually do?

During training it randomly zeros a fraction p of a layer's units each step, so no unit can rely on specific others. This prevents co-adaptation and overfitting, and behaves like training an ensemble of sub-networks.

Why scale by 1/(1 − p)?

Dropping units lowers the layer's output; dividing survivors by (1 − p) makes the expected output equal the full network's, so no rescaling is needed at test time. This is "inverted dropout."

Is dropout used at test time?

No. Dropout is off during inference — the full network runs. (The exception is Monte Carlo dropout, where it's deliberately kept on to estimate uncertainty.)

Do transformers use dropout?

Yes, usually lightly — small dropout on attention and feed-forward layers — though large models lean more on normalization, weight decay, and sheer data for regularization.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

Dropout,
explained.

The co-adaptation problem

The trick: drop units at random

Why the survivors scale up

An ensemble of 2ⁿ networks

Dropout, and the expectation it preserves

In practice

What it left behind

Quick answers

What does dropout actually do?

Why scale by 1/(1 − p)?

Is dropout used at test time?

Do transformers use dropout?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Dropout,explained.

The co-adaptation problem

The trick: drop units at random

Why the survivors scale up

An ensemble of 2ⁿ networks

Dropout, and the expectation it preserves

In practice

What it left behind

Quick answers

What does dropout actually do?

Why scale by 1/(1 − p)?

Is dropout used at test time?

Do transformers use dropout?

Go deeper on training

Adam →

ResNet →

AlexNet →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

Dropout,
explained.