ZeRO / FSDP
also: ZeRO · Fully Sharded Data Parallel · FSDP
Sharding optimizer state, gradients, and even parameters across GPUs instead of replicating them on every one.
Plain data parallelism wastes memory: every GPU stores a full copy of the optimizer state, gradients, and weights. ZeRO (and PyTorch’s implementation, FSDP) shards these across GPUs instead, gathering only the piece a device needs, when it needs it, then releasing it — trading extra communication for dramatically lower per-GPU memory, which is what lets you train models far bigger than any single GPU’s memory using ordinary data-parallel-style training.