Gradient Clipping
Capping the size of the gradient before applying it, so one unusually large batch can’t blow up training.
Gradient clipping rescales the gradient if its norm exceeds a threshold, capping how large a single weight update can be. Without it, a rare batch or an unstable region of the loss landscape can produce an enormous gradient that destabilizes training for many steps afterward — clipping is cheap insurance that’s standard in nearly every large-scale training run.
Worked example: if a gradient’s norm exceeds a threshold, scale it down so no single step blows up the weights — a cheap guard against the exploding-gradient instability that spikes the loss to NaN. Gotcha: clip too aggressively and you slow learning by discarding real signal; the threshold is a hyperparameter, and a run that only survives with a very low clip is often masking a deeper instability.