Data Parallelism
Copying the whole model onto every GPU and splitting the batch across them, syncing gradients each step.
Data parallelism replicates the full model on each GPU, feeds each replica a different slice of the batch, and averages gradients across replicas (via all-reduce) before every weight update — so all copies stay identical. It’s the simplest way to scale training throughput, but every GPU needs enough memory to hold the whole model, which is exactly what ZeRO/FSDP relaxes.
Worked example: each GPU holds a full copy of the model and processes a different slice of the batch; after the backward pass, gradients are averaged across GPUs (all-reduce) so replicas stay in sync. Gotcha: it scales throughput but not model size — the whole model must fit on one GPU — and the gradient all-reduce is a communication cost that bottlenecks at scale, which is why big models add tensor/pipeline parallelism.