Layer Normalization
also: LayerNorm · RMSNorm
Rescaling a layer’s activations to a stable range on every forward pass, so deep networks train without exploding or vanishing.
Layer normalization normalizes activations across the feature dimension for each individual example, keeping their scale stable regardless of what the previous layer produced — critical for training very deep networks like Transformers without gradients exploding or vanishing. RMSNorm, used in most modern LLMs, simplifies this by dropping the mean-centering step and normalizing only by scale, for a small speed win with little quality cost.