Transformers power essentially every modern language model, and the one idea that makes them work is self-attention: a mechanism that lets each token in a sequence look at every other token and pull in the information it needs. Here’s the whole trick, with no hand-waving. Each token is turned into three vectors by learned projections — a query (what am I looking for?), a key (what do I offer?), and a value (what will I contribute?). To update one token, you take its query and compare it against every token’s key with a dot product, producing a score for each — how relevant that token is. You scale the scores and run them through a softmax, turning them into attention weights that sum to one. Finally, the token’s new representation is the weighted sum of every token’s value, dominated by whichever tokens it attended to most. This is how a model lets the word “it” reach back and attend to “robot,” or a verb attend to its subject — relationships computed, not hard-coded. Pick a query token below and watch its scores turn into weights turn into a context vector, with the real arithmetic on show.
query · key · value · scores = Q·K / √d · softmax → weights · output = weighted sum of values
query / key / value
Three vectors per token from learned projections: query = what I seek, key = what I offer, value = what I contribute.
score
The dot product of one token’s query with another’s key — how relevant that token is, before scaling.
softmax
Turns the raw scores into attention weights that are positive and sum to one — a probability-like distribution over tokens.
context vector
A token’s new representation: the weighted sum of every token’s value, using the attention weights.
attention.js — scores → softmax → weighted sum
Ready
A four-token sequence: The robot picked it. Pick a query token to see how it attends to every token. Step through the three stages: raw scores (query·key), the softmax weights, and the resulting context vector. Watch “it” attend to “robot.”
it
query token
robot
attends most to
0.43
top weight
How it works
Self-attention is often wrapped in intimidating notation, but every symbol maps to a concrete step you just watched. For a sequence, the model has three learned weight matrices that project each token’s embedding into a query, a key, and a value vector. The famous formula Attention(Q,K,V) = softmax(QKᵀ / √d) · V is exactly the pipeline: QKᵀ is every query dotted with every key — an all-pairs relevance matrix of scores; dividing by √d (the key dimension) keeps the scores from growing large enough to saturate the softmax and kill its gradients; softmax turns each query’s row of scores into attention weights that sum to one; and multiplying by V replaces each token with the weighted sum of the values it attended to. Three properties make this so powerful. First, it’s content-based: which tokens interact is decided by the learned query/key vectors, so a model can learn that pronouns should attend to their referents or that a verb should attend to its subject — relationships, not fixed positions. Second, every token attends to every other in one step, so information crosses the whole sequence in a single layer (unlike a recurrent network that must pass it along position by position) — and because all the dot products are independent, they run in parallel on a GPU, which is the practical reason transformers scale. Third, real transformers run several attention “heads” in parallel (each with its own Q/K/V projections, so different heads can capture different relationships) and stack many such layers with feed-forward networks in between, so representations get progressively richer. The cost is that all-pairs attention is quadratic in sequence length — the reason long-context efficiency is such an active research area — but the core computation is exactly the scores-to-weights-to-weighted-sum you drove above.
1
Project into query, key, value
Each token’s embedding is turned into three vectors by learned matrices: a query (what it’s looking for), a key (what it offers), and a value (what it will contribute if attended to).
2
Score: query · every key
For the chosen token, dot its query with every token’s key to get a raw relevance score per token, then scale by √d so the softmax stays well-behaved. Higher score = more relevant.
3
Softmax → attention weights
Run the scores through a softmax so they become positive weights summing to one — a distribution over which tokens this token is paying attention to. This is where “it” concentrates its weight on “robot.”
✓
Weighted sum of values → context
The token’s new representation is the weighted sum of every token’s value vector, using those weights — dominated by whatever it attended to. Do this for every token, in parallel, stacked over many heads and layers: that’s a transformer.
Per token
query, key, value
Score
query · key / √d
Weights
softmax(scores)
Output
Σ weightᵢ · valueᵢ
The code
# Self-attention for one query token (the core of a transformer)defattention(query, keys, values, d):
scores = [dot(query, k) / sqrt(d) for k in keys] # query · every key
weights = softmax(scores) # sum to 1, positive
context = sum(w * v for w, v in zip(weights, values))
return context # weighted sum of values# The whole layer, vectorized:# Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V# run in parallel across all tokens, several heads, many layers.
Quick check
1. What are the query, key, and value vectors for a token?
Each token’s embedding is projected by three learned weight matrices into a query, a key, and a value. The query says what the token is looking for, the key says what it offers to others, and the value is what it contributes to any token that attends to it.
2. How is the attention score between two tokens computed?
The score is the dot product of the query with the key, divided by √d (the key dimension) to keep the values from saturating the softmax. A high dot product means the two vectors point in similar directions — the token is highly relevant.
3. What is a token’s output (context vector) after attention?
After softmax gives weights that sum to one, the token’s new representation is the weighted sum of all the value vectors. It’s dominated by whichever tokens received the most attention weight, but every token contributes something — that’s how information flows between positions.
FAQ
What is self-attention in a transformer?
The mechanism that lets each token gather information from every other token. Each token is projected into a query, key, and value; a token’s query is dotted against every key to get relevance scores, which are scaled and softmaxed into weights summing to one, and the token’s new vector is the weighted sum of all values. This learns content-based relationships (like a pronoun attending to its referent) and is repeated across multiple heads and stacked layers to build a transformer.
Why divide the attention scores by the square root of the dimension?
As the query/key dimension grows, the dot-product scores tend to grow in magnitude (more terms summed). Large scores make the softmax extremely peaked, where its gradients become tiny and learning stalls. Dividing by √d rescales the scores to a moderate range so the softmax stays smooth and trainable — a small normalization with an outsized effect on stability, which is why it’s called scaled dot-product attention.
What is multi-head attention and why use several heads?
Multi-head attention runs several attention computations in parallel, each with its own query/key/value projections, then concatenates and projects the results. A single attention pattern captures one kind of relationship at a time; multiple heads let different heads specialize — subject–verb agreement, pronoun resolution, nearby words — at once. Each head works in a lower-dimensional subspace, so total cost is similar to one full attention while greatly enriching what the model can represent.
Why are transformers more parallelizable than RNNs?
An RNN processes a sequence one position at a time, carrying a hidden state forward, so it’s inherently sequential and slow on long sequences. Self-attention lets every token attend to every other in a single step, and all those dot products are independent — so they run simultaneously as big matrix multiplications on a GPU. That parallelism is the main reason transformers replaced RNNs for large-scale training. The trade-off: all-pairs attention is quadratic in sequence length, which is why efficient long-context attention is actively researched.