Distributed training: when the model doesn't fit, split it four ways.

Before splitting anything, know what you're splitting. Training an N-parameter model with Adam in mixed precision holds, per parameter: FP16 weights (2 bytes), FP16 gradients (2), and the optimizer's FP32 master weights + two moments (12) — roughly 16-20 bytes per parameter, before a single activation. A 70B model is ~1.2 TB of state; the biggest GPU holds 80-140 GB. Plus activations — the forward-pass values stashed for the backward — which scale with batch size and sequence length and often rival the weights.

The workhorse. Every GPU holds a full model copy and processes a different batch slice; after each backward pass, an all-reduce averages gradients across all GPUs so the copies stay identical. Simple, scales throughput near-linearly (until all-reduce saturates the network), and it's the default in every framework.

Its one fatal assumption: the whole model fits on one GPU. The moment it doesn't, pure data parallelism is dead — and every strategy below exists to lift that ceiling. The all-reduce also becomes the scaling wall at large GPU counts, which is why the communication-overlap tricks in section 06 matter.

Data parallelism is wildly redundant: every GPU stores the same optimizer states, gradients and weights. ZeRO (DeepSpeed) and FSDP (PyTorch) ask the obvious question — why? — and shard that state across GPUs, gathering each layer's parameters just-in-time for its forward/backward, then releasing them.

The beauty: it stays logically data parallel — nearly the same training code, dramatically higher memory ceiling, at the cost of extra all-gather traffic. For most teams training big-but-not-frontier models, FSDP/ZeRO-3 is the answer, full stop; the exotic strategies below are for when even this isn't enough.

When a single layer's matrices are too big, split within the operation: a matrix multiply is divided column- or row-wise across GPUs, each computes its shard, and they combine results with an all-reduce. This happens inside every layer, on every forward and backward — so tensor parallelism is communication-hungry and lives on NVLink, staying inside a single node (typically 8 GPUs). Cross-node tensor parallelism drowns in interconnect latency. It's the tool for making an individual layer fit, not for scaling out broadly.

Cut the model by layers: GPU 0 runs layers 1-10, GPU 1 runs 11-20, and so on, passing activations down the line like a factory. It moves only activations at stage boundaries — bandwidth-light enough to cross nodes. The catch is the pipeline bubble: while stage 1 works on the first micro-batch, stages 2-4 sit idle waiting for it. The fix is micro-batching — split the batch into many small pieces so all stages stay busy processing different micro-batches at once, shrinking the idle bubble toward zero. Pipeline parallelism is how you scale across the datacenter where tensor parallelism can't reach.

No single axis scales far enough, so frontier training composes all three: tensor parallelism inside a node (hungry, needs NVLink), pipeline parallelism across a handful of nodes (layer stages), data parallelism across the resulting groups (replicas on different data). Each covers another's weakness, mapped deliberately onto the NVLink-fast / InfiniBand-slow topology.

The honest order of operations: reach for mixed precision + FSDP + activation checkpointing first — that combination trains most large models teams actually build. Tensor and pipeline parallelism are the frontier-scale tools you add only when sharding alone runs out of room.