Attention
also: self-attention · scaled dot-product attention
The operation that lets each token look at every other token and weight how much to listen to each.
Attention computes, for every token, a weighted blend of the other tokens’ values — weighted by how much its query matches each key: softmax(QKᵀ/√d)·V. It is the core mechanism of the Transformer, letting the model relate distant words directly instead of passing information step by step.
Worked example: in “the cat sat because it was tired”, the token “it” produces a query that matches the key of “cat” most strongly, so attention routes “cat”’s value into “it” — that is how the model resolves the reference. The √d divisor keeps the dot products from growing with dimension and saturating the softmax. Gotcha: attention is O(n²) in sequence length — double the context and you roughly quadruple the attention compute and memory, which is exactly why long context is expensive and why sparse/linear-attention variants exist.