What is batch normalization?

Batch normalization (Ioffe & Szegedy, 2015) is a layer that, for each feature, normalizes activations across the current mini-batch to zero mean and unit variance, then applies a learnable scale (gamma) and shift (beta). Normalizing keeps the distribution the next layer sees stable during training, which lets networks train much faster and more reliably, tolerate higher learning rates, and be less sensitive to initialization.

Why does batch normalization use learnable gamma and beta?

Forcing every layer's output to be exactly zero-mean and unit-variance would remove expressiveness — some layers need a different scale or a nonzero mean (for example to use the nonlinear part of an activation). The learnable scale gamma and shift beta let the network recover any mean and variance it wants, including undoing the normalization entirely, so no representational power is lost.

How does batch normalization behave differently at test time?

During training it normalizes with the current mini-batch's mean and variance. At test time there may be no batch, and results shouldn't depend on which other examples happen to be present, so it uses running (exponential moving) averages of the mean and variance accumulated during training. Forgetting to switch to these running statistics is a classic source of train/test discrepancies.

Batch norm vs layer norm — what's the difference?

Batch norm normalizes each feature across the examples in a batch, so it depends on batch size and composition. Layer norm normalizes across the features of a single example, so it is independent of the batch — which is why transformers and RNNs, where batch statistics are awkward or vary in sequence length, overwhelmingly use layer normalization instead.

Paper Breakdowns / Batch Normalization

Paper 46~11 min readICML 2015worked math + runnable code

Paper Breakdown

Batch Normalization,
explained.

Before this paper, training a deep network was a superstition-ridden ordeal of careful initialization and tiny learning rates. Then came one deceptively simple layer: at each step, re-center and re-scale a layer's outputs so the next layer always sees a stable distribution. Training sped up, learning rates went up, and depth got easier. Here's the whole transform — and the learnable knob that keeps it from throwing away information.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

The shifting-target problem Normalize across the batch The learnable scale & shift Train vs test statistics Run it yourself Why it helps so much Batch norm's descendants

The shifting-target problem

A deep network is a stack of layers, each learning to transform the outputs of the one below. But those outputs are a moving target: as the lower layer's weights update, the distribution of what it feeds upward keeps shifting — different mean, different spread, step after step. Every layer is chasing a target that won't hold still. The paper named this internal covariate shift, and argued it forces you to use tiny learning rates and careful initialization to keep the whole tower from diverging.

The fix is direct: don't let the distribution wander. After a layer computes its outputs, normalize them — re-center to zero mean and re-scale to unit variance — before passing them on. Now the layer above always receives inputs in a predictable range, no matter how the layer below drifts, and it can learn against a stable target.

The one-sentence version

Standardize each layer's outputs so the next layer always sees the same well-behaved distribution — then give the network a learnable knob to rescale if it needs to.

Normalize across the batch

The normalization is computed per feature, across the mini-batch. For a given feature, take its values over all examples in the current batch, compute their mean and variance, subtract the mean, divide by the standard deviation. A tiny ε in the denominator guards against a feature that happens to be constant (zero variance) in the batch.

The batch-norm transform, per feature

μ_BMean of this feature over the batch.

σ²_BVariance of this feature over the batch.

x̂Standardize: (x − μ_B) / √(σ²_B + ε) → zero mean, unit variance.

yScale & shift with learned γ, β → the layer's output.

Because the statistics come from the batch, batch norm couples the examples in a batch together — a subtle but important property. It also means very small batches give noisy statistics, which is one of batch norm's real limitations.

The learnable scale and shift

Forcing every layer to output exactly zero-mean, unit-variance activations would be too restrictive — sometimes a layer genuinely needs a different scale, or a nonzero mean to reach the useful part of its activation function. So batch norm adds two learnable parameters per feature, a scale γ and a shift β, applied after standardizing. The full transform:

μ_B = mean(x) σ²_B = var(x) x̂ = (x − μ_B) / √(σ²_B + ε)
y = γ · x̂ + β ⇒ 𝔼[y] = β , Var[y] = γ²

Crucially, this does not cost expressiveness. If the optimal thing is to undo the normalization entirely, the network can learn γ = √(σ²_B + ε) and β = μ_B, recovering the original activations exactly. Normalization sets a stable default the network can override — it removes the difficulty without removing the freedom. The runnable version below confirms both halves: standardizing yields mean≈0/var≈1, and γ, β restore any target mean and standard deviation.

Train vs test statistics

There is a catch at inference. During training, the mean and variance come from the current batch. But at test time you may be classifying a single image — there is no batch — and, more importantly, a prediction should never depend on which other examples happen to share its batch. So batch norm keeps a running average of the mean and variance seen during training (an exponential moving average), and uses those fixed statistics at test time:

running ← (1 − momentum)·running + momentum·batch_stat

This is the single most common batch-norm bug in practice: run the model in "training" mode at evaluation and it will normalize with batch statistics — leaking information across test examples and quietly wrecking your metrics. Every framework has an explicit train/eval switch precisely because of this layer.

RUN IT YOURSELF

The transform, and what it preserves

The whole batch-norm transform in a handful of functions, with its two guarantees made checkable. Standardizing a batch gives mean ≈ 0 and variance ≈ 1; the normalization is invariant to any rescaling or shifting of the inputs (so the next layer is shielded from the previous layer's drift); the learnable γ and β restore an arbitrary mean and standard deviation (γ=5, β=3 → std 5, mean 3), proving no expressiveness is lost; the running-stat EMA is what test time uses; and ε keeps a constant feature finite. Change the batch, γ, β, or ε and watch it hold.

CPython · WebAssembly

import math

# Batch norm: normalize each feature across the batch to mean 0 / variance 1,
# then scale+shift by LEARNED gamma, beta. Normalizing stabilizes what the next
# layer sees; gamma/beta return the freedom to be any scale (even the identity).

def mean(xs): return sum(xs) / len(xs)
def var(xs):
    m = mean(xs)
    return sum((x - m) ** 2 for x in xs) / len(xs)   # population variance, as BN uses

def normalize(batch, eps=1e-5):
    m, v = mean(batch), var(batch)
    return [(x - m) / math.sqrt(v + eps) for x in batch]

def bn_forward(batch, gamma, beta, eps=1e-5):
    return [gamma * xh + beta for xh in normalize(batch, eps)]

def ema(running, batch_stat, momentum=0.1):          # test-time running stats
    return (1 - momentum) * running + momentum * batch_stat

batch = [2.0, 4.0, 6.0, 8.0, 10.0]
z = normalize(batch)
print(f"normalized mean / var : {mean(z):.2e} / {var(z):.6f}   (~0 and ~1)")

shifted = [3 * x + 7 for x in batch]                 # rescale + shift the whole batch
print(f"scale+shift invariance: max delta {max(abs(a - b) for a, b in zip(z, normalize(shifted))):.2e}\n")

out = bn_forward(batch, gamma=5.0, beta=3.0)
print(f"gamma=5, beta=3       : mean {mean(out):.4f}, std {math.sqrt(var(out)):.4f}   (BN restores any scale)")
print(f"constant feature ok?  : {normalize([5.0, 5.0, 5.0])}   (epsilon avoids /0)")

Why it helps so much

The measured effects were dramatic — the paper reached the same ImageNet accuracy in a fraction of the training steps — and they compound:

Effect	Why it happens
Faster training	Stable input distributions let gradients flow cleanly; fewer steps to converge.
Higher learning rates	Normalization bounds activation scale, so large steps don't blow up — a major speedup.
Less init-sensitivity	The layer re-centers regardless of how weights were initialized, forgiving bad starts.
Mild regularization	Batch statistics add noise per example, a small dropout-like effect (sometimes reducing the need for dropout).

Interestingly, the original "internal covariate shift" explanation is now debated — later work argued batch norm mainly helps by smoothing the loss landscape, making the optimization surface less jagged. The mechanism is still studied; that it works is not in doubt.

Batch norm's descendants

Batch norm's dependence on the batch is also its weakness: tiny batches give noisy statistics, and sequence models or distributed training make batch statistics awkward. That spawned a family of alternatives that normalize over different axes. Layer norm normalizes across a single example's features (no batch dependence) — which is why it, not batch norm, is standard in transformers and RNNs. Group norm and instance norm split the difference for vision, and RMSNorm strips layer norm down further for large language models.

But the core insight is universal and permanent: keeping activations at a controlled scale, with a learnable way to opt out, makes deep networks trainable. Nearly every modern architecture has a normalization layer of some kind in every block, and every one of them descends from this paper.

Worth knowing

At inference, batch norm's linear transform can be folded into the preceding convolution or linear layer's weights — so it costs literally nothing at test time, a common deployment optimization.

Frequently asked

Quick answers

What does batch norm normalize over?

Each feature across the examples in the mini-batch — subtract the batch mean, divide by the batch standard deviation, then apply learnable scale γ and shift β.

Why the learnable γ and β?

So normalization doesn't cost expressiveness: the network can rescale or even fully undo the standardization (γ = std, β = mean) if that's what it needs.

What changes at test time?

It uses running (moving-average) mean and variance from training instead of batch statistics, so predictions don't depend on the batch. Forgetting to switch modes is a classic bug.

Why do transformers use layer norm instead?

Layer norm normalizes across a single example's features, independent of batch size and sequence length — a better fit than batch norm for sequence models.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

Batch Normalization,
explained.

The shifting-target problem

Normalize across the batch

The learnable scale and shift

Train vs test statistics

The transform, and what it preserves

Why it helps so much

Batch norm's descendants

Quick answers

What does batch norm normalize over?

Why the learnable γ and β?

What changes at test time?

Why do transformers use layer norm instead?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Batch Normalization,explained.

The shifting-target problem

Normalize across the batch

The learnable scale and shift

Train vs test statistics

The transform, and what it preserves

Why it helps so much

Batch norm's descendants

Quick answers

What does batch norm normalize over?

Why the learnable γ and β?

What changes at test time?

Why do transformers use layer norm instead?

Go deeper on training

ResNet →

Adam →

Dropout →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

Batch Normalization,
explained.