LayerNorm & RMSNorm,
explained.

This rung supersedes rung 2 rather than building on it, and it does so by changing one thing: which numbers get averaged together. Batch norm normalises one feature across all the examples in the batch. LayerNorm normalises one example across all of its features. Same operation, transposed axis — and every consequence follows from the transpose.

Because the statistics now come from within a single example, the layer has no batch dependence at all. A batch of one behaves identically to a batch of 512. Sequences of different lengths are no longer a problem, because nothing is averaged across positions that may or may not exist. And the train/test asymmetry that rung 2 introduced — running-average statistics, and the mode switch you can forget — simply disappears: there are no running statistics to keep, because the layer computes everything it needs from the input in front of it.

RMSNorm is then a further subtraction from LayerNorm rather than an addition to it. It observes that of the two things LayerNorm does — re-centre to zero mean, then re-scale to unit variance — the re-centring can be dropped, leaving only division by the root mean square. That removes one reduction over the feature dimension and the mean-subtraction itself, and it changes the invariance the layer provides: LayerNorm's output is unchanged if you shift and scale the input, RMSNorm's only if you scale it. The claim is that the scale invariance was doing the work and the shift invariance was along for the ride.

Batch normalization computes its statistics across the examples in a mini-batch. That is fine for a stack of images, but it breaks down for language. Sequences vary in length; batches are often small; and in autoregressive generation you process one token at a time — there is no batch to average over. Worse, batch statistics make one example's output depend on which others happen to share its batch, which is a strange property for a model that should treat each input on its own terms.

Layer normalization sidesteps all of it with one change of axis: instead of normalizing each feature across examples, normalize each example across its own features. Now the computation involves only the single example in front of it — identical at train and test time, indifferent to batch size, perfect for sequences.

For a single feature vector x (say the hidden state at one token position), layer norm computes the mean and variance over that vector's own components, standardizes, then applies a learnable per-feature scale γ and shift β — the same γ/β for every position, learned during training.

The shape of the transform is identical to batch norm's; only the axis of the statistics moved. That single move is why it slots cleanly into transformers — it appears before or after each attention and feed-forward sub-layer (the "pre-norm" vs "post-norm" choice), keeping activations at a controlled scale block after block, all the way down a hundred-layer stack.

RMSNorm asked whether the mean-centering earns its keep. Its answer: mostly no. It removes the mean subtraction and, in its basic form, the learnable shift β (the paper keeps an optional bias b), and simply rescales the vector by its root-mean-square. Side by side:

RMSNorm keeps the useful part — bounding the vector's magnitude so the next layer sees a controlled scale — and drops the centering. It computes one statistic instead of two and skips a subtraction across the whole hidden dimension, which at billions of parameters and trillions of tokens is a real saving. In practice it matches LayerNorm's quality, so most recent LLMs (LLaMA, and many since) use it. The runnable version below shows the exact consequence of dropping the mean — and the case where it makes no difference at all.

The mathematical difference between the two is captured entirely by their invariances — what you can do to the input without changing the output. Both are invariant to rescaling the input vector (multiply every component by a constant and the normalization cancels it). But only LayerNorm is invariant to shifting — adding a constant to every component — because only LayerNorm subtracts the mean.

That last row is the punchline: RMSNorm is precisely LayerNorm with the centering deleted, so on any already-centered input they compute the very same thing. The empirical success of RMSNorm is essentially the finding that, for deep transformers, the shift-invariance LayerNorm provided was not doing much load-bearing work.

LayerNorm (Ba, Kiros and Hinton, 2016)

1. Take one layer's summed inputs for one example. For a layer of H hidden units, a is the vector of pre-activations before the nonlinearity.
2. Compute two statistics across the layer. μ = (1/H)Σa_i and σ = √((1/H)Σ(a_i − μ)²). Every unit in the layer shares them, and they depend on nothing outside this example.
3. Normalize, then apply a per-unit gain and bias. h = f(g/σ ⊙ (a − μ) + b), with the gains initialized to 1 and the biases to 0.
4. In a recurrent net, recompute at every step. With a^^t = W_hhh^^t−1 + W_xhx^^t, the statistics come fresh from a^^t at each time step while g and b are shared across time. Batch norm in an RNN has to store separate statistics for each time step, which fails for a test sequence longer than any training sequence; layer norm has no such problem. For LSTMs the authors normalize the recurrent and input contributions to the gates separately, and normalize the cell state before the output tanh.
5. Do exactly the same at test time. No running averages are kept.

The paper's analysis compares invariances. Normalizing over a layer leaves the output unchanged when the whole weight matrix is rescaled or re-centered (the same vector added to every unit's incoming weights), and when a single training case is rescaled, but not when one unit's weight vector is rescaled, which batch norm and weight normalization tolerate. Analyzing the Fisher information of a normalized generalized linear model, the authors argue that as a weight vector's norm grows, σ grows with it and the curvature in that direction falls, so the effective learning rate drops on its own, an implicit early-stopping effect, and that learning the gain depends only on the size of the prediction error, not on how the inputs are scaled.

RMSNorm (Zhang and Sennrich, 2019)

1. Keep only the scale statistic. RMS(a) = √((1/n)Σa_i²), and each output is a_i/RMS(a) times a gain g_i. The paper's layer form still allows a bias, y = f(Wx/RMS(a) ⊙ g + b).
2. Keep the invariance that matters. Because RMS scales linearly with its argument, the layer is invariant to rescaling both its input and its weight matrix, but not to re-centering; the paper's claim is that LayerNorm's re-centering invariance is dispensable. Its gradient analysis shows the weight gradient falls as the weight matrix is scaled up, which the authors describe as implicit learning-rate adaptation.
3. Optionally estimate the RMS from a slice. Partial RMSNorm (pRMSNorm) computes the RMS from only the first k = ⌈n·p⌉ summed inputs, on the reasoning that a sample of the units estimates the layer's scale; the experiments use p = 6.25%.
4. Measure time as well as quality. Every comparison reports wall-clock time per training step next to BLEU, accuracy or recall, across TensorFlow, Theano and PyTorch implementations.

The figure shows the two places either normalizer is put inside a transformer block, a question neither paper addresses: after the residual addition (post-norm, the original transformer) or on the branch before each sublayer (pre-norm).

• a^t — the summed inputs at time t, from the previous hidden state and the current input.
• μ^t, σ^t — statistics over the H units at this step only; nothing is carried across steps or across the batch.
• g, b — one gain and one bias per unit, shared across all time steps.
• k = ⌈n·p⌉ — how many of the n summed inputs the partial RMS reads; with p = 6.25%, a 512-unit layer reads k = 32.

The extremes explain the design. Multiply every a_i^^t by c > 0 and both a − μ and σ scale by c, so h^^t is unchanged: the recurrent dynamics cannot blow up through the scale of the summed inputs, which is the stability the paper claims for RNNs. If all units take the same value, a − μ = 0 and the layer outputs f(b), every distinction erased, while RMSNorm in the same case outputs ±g. At p = 100%, pRMSNorm is RMSNorm; as p shrinks the estimate gets noisier, and the authors report gradients that tend to explode at small ratios.

LayerNorm was tested mainly on recurrent models, each against the published baseline and, where one existed, recurrent batch norm:

RMSNorm was compared directly with LayerNorm on the same models, timed per 1,000 steps (per 100 for retrieval):

The negative results matter as much. Without any normalization the Transformer failed to train, and on images batch norm stayed ahead: on CIFAR-10, LayerNorm was worse even than the unnormalized network (8.96%). Dividing by the plain L2 norm instead of the RMS, which drops the √n scaling, did not improve the RNNSearch model at all (20.7 BLEU against 21.7 with no normalization). And pRMSNorm's theoretical saving did not show up consistently in wall-clock time, which the authors blame on the cost of slicing tensors.

The LayerNorm paper's own results are honest in a way that reads oddly now: its gains are demonstrated on recurrent networks and on fully-connected models, and it reports that for convolutional networks batch norm remained the better choice. At the time that was the main event and the sequence-model result was the side note. The transformer inverted the importance of those two findings three years later.

What neither paper settles is where the normaliser belongs relative to rung 4's skip connection, and the answer turned out to matter more than the choice between the two normalisers. The original transformer applied it after the residual addition; that configuration needs a learning-rate warmup to train at all, and Xiong and colleagues showed in 2020 that moving it inside the block, before the sublayer, makes training markedly more stable and the warmup less critical. Pre-norm is now the default, and the reason is that it leaves the residual stream itself unnormalised — an uninterrupted path from input to output.

How it held up: LayerNorm won the sequence-model argument completely, and then RMSNorm largely won the argument against LayerNorm on cost grounds — it is what sits inside LLaMA and most of the open-weight models on that ladder. The honest summary of the gap between them is that the quality difference is small and the speed difference is real, which is why the cheaper one spread. Batch norm did not lose; it stayed where its assumption holds, which is vision models with fixed-shape inputs and a batch to spare.

Open the source of any modern language model and you will find a normalization layer at the top of every block — almost always LayerNorm or RMSNorm. They are not glamorous, but they are load-bearing: without them, gradients through a deep pre-norm stack would not stay well-scaled, and training the models we take for granted would be far harder. Layer norm is one of the three or four ideas that made the transformer trainable at scale.

RMSNorm's arc is a small lesson in engineering, too. It won not by adding capability but by removing a piece — the mean subtraction — that careful measurement showed was carrying little weight. Faster and simpler, at no measured cost in quality: exactly the kind of subtraction that compounds when you are training a model a trillion tokens at a time.

Normalization is one of those unglamorous pieces without which deep networks simply don't train. As signals pass through many layers, their scale drifts — exploding or vanishing — and gradients go with it. LayerNorm fixes this by normalizing each token's activations to zero mean and unit variance (then rescaling with learned gain/bias), keeping the numbers in a sane range at every layer so optimization stays stable.

LayerNorm won in Transformers over BatchNorm for a specific reason: it normalizes across the features of a single token, independent of the other examples in the batch. That makes it insensitive to batch size and to variable sequence lengths — exactly the conditions language models train and serve under, where BatchNorm's cross-example statistics are awkward or unavailable.

RMSNorm is the modern simplification: it drops the mean-centering and the bias, normalizing only by the root-mean-square of the activations. It turns out the re-centering contributes little, so RMSNorm gets essentially the same stabilizing benefit with less computation — which is why many recent large models adopt it. A small efficiency win, repeated across every layer and every token, adds up.

The other detail that matters in practice is where you normalize: pre-norm (normalize before each sublayer) versus post-norm. Pre-norm gives cleaner gradient flow and is what makes very deep Transformers trainable, so it became standard. The lasting lesson is that these "boring" normalization choices are load-bearing — get them wrong and a large model won't train at all.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →