Layer normalization normalizes activations across the feature dimension for each individual example, keeping their scale stable regardless of what the previous layer produced — critical for training very deep networks like Transformers without gradients exploding or vanishing. RMSNorm, used in most modern LLMs, simplifies this by dropping the mean-centering step and normalizing only by scale, for a small speed win with little quality cost.
Worked example: normalizes each token’s activation vector to zero mean and unit variance across its features (then rescales with learned parameters), stabilizing training by keeping activations well-behaved regardless of depth. Gotcha: unlike batch-norm it is independent of batch size (normalizes per-token, not across the batch), which is why transformers use it — and WHERE matters: pre-norm (before the sublayer) trains more stably at depth than the original post-norm, so modern transformers moved to pre-norm.