Pipeline Parallelism
Splitting a model’s layers across GPUs, so one device holds the early layers and another holds the later ones.
Pipeline parallelism assigns different consecutive layers of a model to different GPUs, so a forward pass flows through devices in sequence. Naively this leaves most GPUs idle waiting for the previous one — the pipeline bubble — which is why real implementations split each batch into many micro-batches and stream them through the pipeline to keep every stage busy.