Multi-Head Attention
Running several attention operations in parallel, each learning to focus on a different kind of relationship.
Multi-head attention runs several attention operations (“heads”) in parallel, each with its own learned projections, then concatenates their outputs. Each head can specialize — one tracks syntactic dependencies, another coreference, another position — so the model attends to different relationships at once instead of averaging them into one.
Worked example: in a sentence, one head might link a pronoun to its antecedent while another links a verb to its subject, in the same layer — richer than a single attention that must cram all relationships into one weighting. Gotcha: more heads is not strictly better — research shows many heads are redundant and can be pruned with little loss — and the total attention dimension is split across heads (more heads means a smaller per-head dimension), so it is a partition of capacity, not free extra capacity.