Data Parallelism
Copying the whole model onto every GPU and splitting the batch across them, syncing gradients each step.
Data parallelism replicates the full model on each GPU, feeds each replica a different slice of the batch, and averages gradients across replicas (via all-reduce) before every weight update — so all copies stay identical. It’s the simplest way to scale training throughput, but every GPU needs enough memory to hold the whole model, which is exactly what ZeRO/FSDP relaxes.