Plain data parallelism wastes memory: every GPU stores a full copy of the optimizer state, gradients, and weights. ZeRO (and PyTorch’s implementation, FSDP) shards these across GPUs instead, gathering only the piece a device needs, when it needs it, then releasing it — trading extra communication for dramatically lower per-GPU memory, which is what lets you train models far bigger than any single GPU’s memory using ordinary data-parallel-style training.
Worked example: ZeRO (DeepSpeed/FSDP) shards optimizer states, gradients, and even parameters across data-parallel GPUs instead of replicating them, cutting per-GPU memory dramatically so much larger models fit. Gotcha: the savings cost more communication to gather shards when needed; the ZeRO stages (1/2/3) trade progressively more communication for less memory, so pick the stage your interconnect can afford.