Model Parallelism
The umbrella term for splitting one model’s compute across multiple devices — tensor parallelism and pipeline parallelism are its two main forms.
Model parallelism is the general strategy of spreading a single model too large for one GPU across several — as opposed to data parallelism, which spreads the batch instead. In practice, large training runs combine several forms at once (often called 3D parallelism): tensor parallelism within a node, pipeline parallelism across nodes, and data parallelism across the whole cluster.
Worked example: splitting one model across multiple devices because it does not fit on a single GPU — the umbrella for tensor parallelism (split each layer’s matrices), pipeline parallelism (split by layers), and expert parallelism (split MoE experts). Gotcha: unlike data parallelism (which replicates the model for throughput), model parallelism is about FITTING the model, and every split adds cross-device communication inside the forward/backward pass — so it is bounded by interconnect speed and usually combined into 3D parallelism tuned to the hardware.