Batch Normalization,
explained.

Dropout (rung 1) accepts the optimisation problem as given and adds noise to stop the network memorising. Batch norm changes the optimisation problem itself. Every layer in a deep network is trying to learn a function of its input, and that input is the output of the layer below, which is still changing — so each layer is fitting a target that keeps moving as the layers beneath it update.

The change: normalise each feature to zero mean and unit variance across the examples in the mini-batch, as a layer, inside the network, on every forward pass. Because forcing that distribution would also destroy what the layer can express, it then applies a learned scale γ and shift β per feature — so the layer can recover any distribution it wants, including the unnormalised one, but it has to ask for it.

Stated in rung 1's units: dropout adds one hyperparameter and zero parameters; batch norm adds two parameters per feature (γ and β) and, more importantly, buys back a hyperparameter you were already spending — the learning rate. Normalised activations let you raise it by an order of magnitude without diverging, which is where most of the reported speed-up comes from. It also makes initialisation much less delicate, and it regularises hard enough that many recipes dropped rung 1 entirely when they adopted it.

A deep network is a stack of layers, each learning to transform the outputs of the one below. But those outputs are a moving target: as the lower layer's weights update, the distribution of what it feeds upward keeps shifting — different mean, different spread, step after step. Every layer is chasing a target that won't hold still. The paper named this internal covariate shift, and argued it forces you to use tiny learning rates and careful initialization to keep the whole tower from diverging.

The fix is direct: don't let the distribution wander. After a layer computes its outputs, normalize them — re-center to zero mean and re-scale to unit variance — before passing them on. Now the layer above always receives inputs in a predictable range, no matter how the layer below drifts, and it can learn against a stable target.

The normalization is computed per feature, across the mini-batch. For a given feature, take its values over all examples in the current batch, compute their mean and variance, subtract the mean, divide by the standard deviation. A tiny ε in the denominator guards against a feature that happens to be constant (zero variance) in the batch.

Because the statistics come from the batch, batch norm couples the examples in a batch together — a subtle but important property. It also means very small batches give noisy statistics, which is one of batch norm's real limitations.

Forcing every layer to output exactly zero-mean, unit-variance activations would be too restrictive — sometimes a layer genuinely needs a different scale, or a nonzero mean to reach the useful part of its activation function. So batch norm adds two learnable parameters per feature, a scale γ and a shift β, applied after standardizing. The full transform:

Crucially, this does not cost expressiveness. If the optimal thing is to undo the normalization entirely, the network can learn γ = √(σ²_B + ε) and β = μ_B, recovering the original activations exactly. Normalization sets a stable default the network can override — it removes the difficulty without removing the freedom. The runnable version below confirms both halves: standardizing yields mean≈0/var≈1, and γ, β restore any target mean and standard deviation.

There is a catch at inference. During training, the mean and variance come from the current batch. But at test time you may be classifying a single image — there is no batch — and, more importantly, a prediction should never depend on which other examples happen to share its batch. So batch norm keeps a running average of the mean and variance seen during training (an exponential moving average), and uses those fixed statistics at test time:

This is the single most common batch-norm bug in practice: run the model in "training" mode at evaluation and it will normalize with batch statistics — leaking information across test examples and quietly wrecking your metrics. Every framework has an explicit train/eval switch precisely because of this layer.

Ioffe and Szegedy add the transform as a layer between a linear map and its nonlinearity, and they pin down five things about how it runs.

1. Normalize the pre-activation, not the layer input. For z = g(Wu + b), the layer becomes z = g(BN(Wu)). The input u is the output of another nonlinearity and its distribution keeps changing shape, while Wu is more likely to be symmetric, non-sparse and close to Gaussian, so fixing its first two moments gives a steadier distribution. The bias b is dropped, because mean subtraction would cancel it; β takes over its role.
2. Compute statistics per activation over the mini-batch. For each dimension k, the mean μ_B and variance σ²_B are taken over the m examples of the mini-batch, each value is standardized with a small ε added to the variance, and a learned pair γ^^(k), β^^(k) scales and shifts the result.
3. In convolutional layers, pool over space too. To keep the convolutional property that every location of a feature map is treated alike, one channel's mean and variance are computed jointly over the batch and all spatial positions: m examples with p × q maps give an effective batch of m′ = m·p·q values per channel, and γ and β are learned per feature map rather than per activation.
4. Backpropagate through the statistics. μ_B and σ²_B depend on every example in the batch, so the gradient for one example depends on the others. The paper shows why this is required: if the normalization is treated as a constant outside the gradient step, an update to a bias is undone by the next mean subtraction, neither the output nor the loss changes, and the bias grows without bound.
5. Freeze the layer for inference. After training, the batch statistics are replaced by population statistics: the average of μ_B over many training mini-batches, and the average of σ²_B multiplied by m/(m − 1) to make the variance unbiased. The layer then becomes a fixed linear map per dimension, γ/√(Var[x] + ε) · x + (β − γE[x]/√(Var[x] + ε)), and the paper notes that moving averages can track these statistics while the model trains.

The network it was tested in

The ImageNet experiments use a variant of Inception with 13.6 million parameters: each 5 × 5 convolution is replaced by two stacked 3 × 3 convolutions, the 28 × 28 stage gets a third Inception module, and there are no fully connected layers apart from the final softmax. It was trained with SGD with momentum on mini-batches of 32 in a distributed setup, with BN before every nonlinearity. Inserting BN was only the start; the authors then rebuilt the training recipe around it:

• raise the learning rate, with 5× and 30× variants;
• remove dropout, which sped up training without increasing overfitting;
• cut the L2 weight penalty by a factor of 5;
• decay the learning rate 6 times faster;
• remove local response normalization;
• shuffle training examples within shards so the same examples stop sharing mini-batches, worth about 1% validation accuracy;
• reduce photometric distortions, since a faster-training network sees each image fewer times.

The paper derives the gradient of the loss ℓ through μ_B and σ²_B with the chain rule, in six formulas. Writing g_i = ∂ℓ/∂y_i and collecting terms gives one line per example:

• m·gi — the direct path, the gradient as if the statistics were fixed numbers.
• −Σ gj — the path through μ_B: it subtracts the batch total of the incoming gradient, because shifting every x_i by the same amount cannot change x̂.
• −x̂i Σ gjx̂j — the path through σ²_B: it removes the part of the gradient that points along x̂, because stretching all deviations equally cannot change x̂ either.
• γ/√(σ²B + ε) — the same rescaling the forward pass applied.

At the extremes: if every g_j is equal, the bracket is zero and nothing reaches x. With m = 1, x − μ_B = 0, the layer outputs β whatever its input and the gradient to x vanishes, which is why a batch of one cannot train it. The paper adds a scale property, BN((aW)u) = BN(Wu) with ∂BN((aW)u)/∂(aW) = (1/a)·∂BN(Wu)/∂W: larger weights receive proportionally smaller gradients.

A first test on MNIST used three fully connected hidden layers of 100 sigmoid units, trained for 50,000 steps on mini-batches of 60: the BN network reached higher test accuracy, and the inputs to its sigmoids kept a stable distribution through training. The main test is single-crop ImageNet validation accuracy for the Inception variant, measured against the steps the original needed to reach its best:

Adding BN alone cut the steps by more than half; the fourteenfold saving needed the higher learning rate and the other recipe changes. BN-x30 starts slower than BN-x5 but keeps improving, reaching 74.8% after about 6 million steps. The sigmoid run is the sharpest ablation: with BN it trains to 69.8%, while the same sigmoid network without BN never exceeded one-in-a-thousand accuracy, which is chance for 1,000 classes.

A single BN network scored 7.82% top-5 validation error with one crop and 5.82% with many crops; an ensemble of six reached 4.9% on validation and 4.82% on the test server, which the authors note beats the reported accuracy of human raters. Even that ensemble kept a little dropout: some members used 5% or 10%, against 40% in the original Inception.

The paper is explicit about the ugliest consequence, and it is the one that bites in production: the layer behaves differently at training and at inference. At training time the statistics come from the batch; at inference there is no batch, so the layer uses running averages accumulated during training. Forgetting to switch modes is a defect that produces a model which scores well in the training loop and badly when served, with no error message.

What the paper claims and does not establish is the mechanism. It attributes the gain to reducing "internal covariate shift" — the drift in each layer's input distribution. That explanation is contested: Santurkar and colleagues argued in 2018 that covariate shift is not what batch norm fixes, and demonstrated networks where the shift is deliberately reintroduced after normalisation and training is fast anyway; their account is that it smooths the loss surface. The layer works; the stated reason for why it works is not settled.

How it held up: it made very deep convolutional networks routine, and rung 4 is hard to imagine landing without it. But its dependence on the batch is a structural limit, not a detail — with a batch of one, or with sequences of different lengths where "the same feature across the batch" is not a well-defined set of numbers, the statistics are either unavailable or meaningless. That limit is precisely why rung 5 exists, and why the normaliser inside every transformer is not this one.

The measured effects were dramatic — the paper reached the same ImageNet accuracy in a fraction of the training steps — and they compound:

Interestingly, the original "internal covariate shift" explanation is now debated — later work argued batch norm mainly helps by smoothing the loss landscape, making the optimization surface less jagged. The mechanism is still studied; that it works is not in doubt.

Batch norm's dependence on the batch is also its weakness: tiny batches give noisy statistics, and sequence models or distributed training make batch statistics awkward. That spawned a family of alternatives that normalize over different axes. Layer norm normalizes across a single example's features (no batch dependence) — which is why it, not batch norm, is standard in transformers and RNNs. Group norm and instance norm split the difference for vision, and RMSNorm strips layer norm down further for large language models.

But the core insight is universal and permanent: keeping activations at a controlled scale, with a learnable way to opt out, makes deep networks trainable. Nearly every modern architecture has a normalization layer of some kind in every block, and every one of them descends from this paper.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →