Tensor Parallelism
Splitting an individual weight matrix across GPUs so one layer’s math runs cooperatively on several devices at once.
Tensor parallelism shards a single large weight matrix (e.g. an attention or feed-forward layer) column- or row-wise across GPUs, so each device holds a slice and the layer’s output is assembled with an all-reduce across devices. It is what makes a model too big to fit on one GPU trainable at all — at the cost of a communication round every layer, which is why it is usually kept within a fast-interconnect node rather than spread across nodes.
Worked example: a single layer’s weight matrix is split across GPUs, so each computes part of the matmul and they combine results — letting a layer too big for one GPU run across several. Gotcha: it needs fast interconnect (NVLink) because GPUs communicate within every layer, so it is usually confined to GPUs in one node; across nodes the communication cost dominates.