Mixed Precision Training
also: fp16 · bf16 · AMP
Running most of training in 16-bit numbers for speed and memory, while keeping a 32-bit master copy of the weights for stability.
Mixed precision training runs the forward/backward pass in a lower-precision format (fp16 or bf16) — half the memory, faster on modern GPU tensor cores — while keeping a full-precision (fp32) master copy of the weights that the small, precise gradient updates actually accumulate into, preventing the update from underflowing to zero in low precision. This alone is a large fraction of the throughput gain in modern large-scale training.