Weight Decay
also: L2 regularization · AdamW
A regularizer that shrinks weights toward zero each step, discouraging over-large parameters and overfitting.
Weight decay is a regularization technique that adds a penalty proportional to the size of the weights (their squared L2 norm) to the loss, so the optimizer keeps weights small unless the data strongly justifies them. Smaller weights mean a simpler function that generalizes better.
Worked example: training with weight decay, each update nudges every weight a little toward zero in addition to the gradient step — so features that barely help get shrunk away, reducing overfitting. Gotcha: classic L2 weight decay and adding an L2 term to the loss are equivalent for plain SGD but NOT for Adam — which is why AdamW “decouples” weight decay from the adaptive gradient, and using the wrong variant is a subtle bug that quietly hurts generalization in transformer training.