LayerNorm & RMSNorm,
explained.

Batch normalization computes its statistics across the examples in a mini-batch. That is fine for a stack of images, but it breaks down for language. Sequences vary in length; batches are often small; and in autoregressive generation you process one token at a time — there is no batch to average over. Worse, batch statistics make one example's output depend on which others happen to share its batch, which is a strange property for a model that should treat each input on its own terms.

Layer normalization sidesteps all of it with one change of axis: instead of normalizing each feature across examples, normalize each example across its own features. Now the computation involves only the single example in front of it — identical at train and test time, indifferent to batch size, perfect for sequences.

For a single feature vector x (say the hidden state at one token position), layer norm computes the mean and variance over that vector's own components, standardizes, then applies a learnable per-feature scale γ and shift β — the same γ/β for every position, learned during training.

The shape of the transform is identical to batch norm's; only the axis of the statistics moved. That single move is why it slots cleanly into transformers — it appears before or after each attention and feed-forward sub-layer (the "pre-norm" vs "post-norm" choice), keeping activations at a controlled scale block after block, all the way down a hundred-layer stack.

RMSNorm asked whether the mean-centering earns its keep. Its answer: mostly no. It removes both the mean subtraction and the learnable shift β, and simply rescales the vector by its root-mean-square. Side by side:

RMSNorm keeps the useful part — bounding the vector's magnitude so the next layer sees a controlled scale — and drops the centering. It computes one statistic instead of two and skips a subtraction across the whole hidden dimension, which at billions of parameters and trillions of tokens is a real saving. In practice it matches LayerNorm's quality, so most recent LLMs (LLaMA, and many since) use it. The runnable version below shows the exact consequence of dropping the mean — and the case where it makes no difference at all.

The mathematical difference between the two is captured entirely by their invariances — what you can do to the input without changing the output. Both are invariant to rescaling the input vector (multiply every component by a constant and the normalization cancels it). But only LayerNorm is invariant to shifting — adding a constant to every component — because only LayerNorm subtracts the mean.

That last row is the punchline: RMSNorm is precisely LayerNorm with the centering deleted, so on any already-centered input they compute the very same thing. The empirical success of RMSNorm is essentially the finding that, for deep transformers, the shift-invariance LayerNorm provided was not doing much load-bearing work.

A quick map of the normalization landscape and when each is the right reach:

Inside a transformer block, the other live decision is placement: post-norm (original) applies the norm after the residual addition; pre-norm applies it inside the residual branch, before the sub-layer. Pre-norm makes very deep stacks far more stable to train, which is why nearly all large models use it — often with RMSNorm as the norm of choice.

Open the source of any modern language model and you will find a normalization layer at the top of every block — almost always LayerNorm or RMSNorm. They are not glamorous, but they are load-bearing: without them, gradients through a deep pre-norm stack would not stay well-scaled, and training the models we take for granted would be far harder. Layer norm is one of the three or four ideas that made the transformer trainable at scale.

RMSNorm's arc is a small lesson in engineering, too. It won not by adding capability but by removing a piece — the mean subtraction — that careful measurement showed was carrying little weight. Faster and simpler, at no measured cost in quality: exactly the kind of subtraction that compounds when you are training a model a trillion tokens at a time.