Model Parallelism
The umbrella term for splitting one model’s compute across multiple devices — tensor parallelism and pipeline parallelism are its two main forms.
Model parallelism is the general strategy of spreading a single model too large for one GPU across several — as opposed to data parallelism, which spreads the batch instead. In practice, large training runs combine several forms at once (often called 3D parallelism): tensor parallelism within a node, pipeline parallelism across nodes, and data parallelism across the whole cluster.