Learning Rate Schedule
also: warmup · cosine decay
Changing the step size during training — usually a short warmup up, then a long decay down — instead of holding it fixed.
Training with too high a learning rate from step one is unstable; too low wastes compute converging slowly. A learning rate schedule typically warms up linearly for a small fraction of training to let the optimizer’s running statistics stabilize, then decays (commonly following a cosine curve) toward a small value as training approaches convergence, where large steps would overshoot a good minimum.