Normally, every layer’s activations from the forward pass are kept in memory until the backward pass needs them to compute gradients. Gradient checkpointing only saves a handful of these (checkpoints), and recomputes the rest on the fly during the backward pass — roughly one extra forward pass of compute in exchange for a large cut in peak memory, often the difference between a model fitting on your GPUs or not.
Worked example: instead of storing every layer’s activations for the backward pass, checkpointing keeps only a few and recomputes the rest on the fly — trading extra compute for much lower memory, so you can fit bigger models or batches. Gotcha: it typically adds ~30% compute (an extra forward pass over recomputed segments); it is a memory-vs-speed dial, valuable exactly when you are memory-bound with spare compute.