Momentum
An optimizer trick: accumulate a running average of past gradients to speed up and smooth training.
Momentum accelerates gradient descent by accumulating a running (exponentially-weighted) average of past gradients and stepping in that direction, rather than using only the current gradient. Like a ball rolling downhill, it builds speed in consistent directions and dampens oscillation in noisy ones.
Worked example: in a long, narrow valley where plain SGD zig-zags across the walls, momentum averages out the sideways bounces and keeps rolling down the valley floor — reaching the minimum in far fewer steps. Gotcha: momentum is baked into modern optimizers (Adam combines it with per-parameter scaling), and its coefficient (commonly ~0.9) is another knob — too high and it overshoots minima, too low and you lose the acceleration; it helps most on ill-conditioned loss surfaces.