Tensor Parallelism
Splitting an individual weight matrix across GPUs so one layer’s math runs cooperatively on several devices at once.
Tensor parallelism shards a single large weight matrix (e.g. an attention or feed-forward layer) column- or row-wise across GPUs, so each device holds a slice and the layer’s output is assembled with an all-reduce across devices. It is what makes a model too big to fit on one GPU trainable at all — at the cost of a communication round every layer, which is why it is usually kept within a fast-interconnect node rather than spread across nodes.