Pipeline Parallelism
Splitting a model’s layers across GPUs, so one device holds the early layers and another holds the later ones.
Pipeline parallelism assigns different consecutive layers of a model to different GPUs, so a forward pass flows through devices in sequence. Naively this leaves most GPUs idle waiting for the previous one — the pipeline bubble — which is why real implementations split each batch into many micro-batches and stream them through the pipeline to keep every stage busy.
Worked example: different layers live on different GPUs (layers 1–8 on GPU0, 9–16 on GPU1…), and micro-batches flow through the pipeline so all stages stay busy. Gotcha: the ‘pipeline bubble’ — GPUs idle while the pipeline fills and drains — wastes compute; more micro-batches shrink the bubble but raise memory, so it is tuned against the other parallelism dimensions.