Attention
also: self-attention · scaled dot-product attention
The operation that lets each token look at every other token and weight how much to listen to each.
Attention computes, for every token, a weighted blend of the other tokens’ values — weighted by how much its query matches each key: softmax(QKᵀ/√d)·V. It is the core mechanism of the Transformer, letting the model relate distant words directly instead of passing information step by step.