Gradient Checkpointing
also: activation checkpointing
Trading extra compute for less memory by recomputing activations during the backward pass instead of storing them.
Normally, every layer’s activations from the forward pass are kept in memory until the backward pass needs them to compute gradients. Gradient checkpointing only saves a handful of these (checkpoints), and recomputes the rest on the fly during the backward pass — roughly one extra forward pass of compute in exchange for a large cut in peak memory, often the difference between a model fitting on your GPUs or not.