Mixed precision training runs the forward/backward pass in a lower-precision format (fp16 or bf16) — half the memory, faster on modern GPU tensor cores — while keeping a full-precision (fp32) master copy of the weights that the small, precise gradient updates actually accumulate into, preventing the update from underflowing to zero in low precision. This alone is a large fraction of the throughput gain in modern large-scale training.
Worked example: compute in 16-bit (fp16/bf16) for speed and memory while keeping a master copy of weights in fp32 for stability — accelerators run 16-bit matmuls much faster. Gotcha: fp16 has a narrow range that underflows gradients, so loss scaling (or bf16’s wider exponent) avoids silent NaNs; bf16 trades mantissa precision for range and is the common default where hardware supports it.