Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

AI · Transformers

Every Token Looks at Every Token.

Transformers power essentially every modern language model, and the one idea that makes them work is self-attention: a mechanism that lets each token in a sequence look at every other token and pull in the information it needs. Here’s the whole trick, with no hand-waving. Each token is turned into three vectors by learned projections — a query (what am I looking for?), a key (what do I offer?), and a value (what will I contribute?). To update one token, you take its query and compare it against every token’s key with a dot product, producing a score for each — how relevant that token is. You scale the scores and run them through a softmax, turning them into attention weights that sum to one. Finally, the token’s new representation is the weighted sum of every token’s value, dominated by whichever tokens it attended to most. This is how a model lets the word “it” reach back and attend to “robot,” or a verb attend to its subject — relationships computed, not hard-coded. Pick a query token below and watch its scores turn into weights turn into a context vector, with the real arithmetic on show.

query · key · value · scores = Q·K / √d · softmax → weights · output = weighted sum of values
query / key / value

Three vectors per token from learned projections: query = what I seek, key = what I offer, value = what I contribute.

score

The dot product of one token’s query with another’s key — how relevant that token is, before scaling.

softmax

Turns the raw scores into attention weights that are positive and sum to one — a probability-like distribution over tokens.

context vector

A token’s new representation: the weighted sum of every token’s value, using the attention weights.

attention.js — scores → softmax → weighted sum
Ready
A four-token sequence: The robot picked it. Pick a query token to see how it attends to every token. Step through the three stages: raw scores (query·key), the softmax weights, and the resulting context vector. Watch “it” attend to “robot.”
it
query token
robot
attends most to
0.43
top weight

How it works

Self-attention is often wrapped in intimidating notation, but every symbol maps to a concrete step you just watched. For a sequence, the model has three learned weight matrices that project each token’s embedding into a query, a key, and a value vector. The famous formula Attention(Q,K,V) = softmax(QKᵀ / √d) · V is exactly the pipeline: QKᵀ is every query dotted with every key — an all-pairs relevance matrix of scores; dividing by √d (the key dimension) keeps the scores from growing large enough to saturate the softmax and kill its gradients; softmax turns each query’s row of scores into attention weights that sum to one; and multiplying by V replaces each token with the weighted sum of the values it attended to. Three properties make this so powerful. First, it’s content-based: which tokens interact is decided by the learned query/key vectors, so a model can learn that pronouns should attend to their referents or that a verb should attend to its subject — relationships, not fixed positions. Second, every token attends to every other in one step, so information crosses the whole sequence in a single layer (unlike a recurrent network that must pass it along position by position) — and because all the dot products are independent, they run in parallel on a GPU, which is the practical reason transformers scale. Third, real transformers run several attention “heads” in parallel (each with its own Q/K/V projections, so different heads can capture different relationships) and stack many such layers with feed-forward networks in between, so representations get progressively richer. The cost is that all-pairs attention is quadratic in sequence length — the reason long-context efficiency is such an active research area — but the core computation is exactly the scores-to-weights-to-weighted-sum you drove above.

1

Project into query, key, value

Each token’s embedding is turned into three vectors by learned matrices: a query (what it’s looking for), a key (what it offers), and a value (what it will contribute if attended to).

2

Score: query · every key

For the chosen token, dot its query with every token’s key to get a raw relevance score per token, then scale by √d so the softmax stays well-behaved. Higher score = more relevant.

3

Softmax → attention weights

Run the scores through a softmax so they become positive weights summing to one — a distribution over which tokens this token is paying attention to. This is where “it” concentrates its weight on “robot.”

✓

Weighted sum of values → context

The token’s new representation is the weighted sum of every token’s value vector, using those weights — dominated by whatever it attended to. Do this for every token, in parallel, stacked over many heads and layers: that’s a transformer.

Per token
query, key, value
Score
query · key / √d
Weights
softmax(scores)
Output
Σ weightᵢ · valueᵢ

The code

# Self-attention for one query token (the core of a transformer) def attention(query, keys, values, d): scores = [dot(query, k) / sqrt(d) for k in keys] # query · every key weights = softmax(scores) # sum to 1, positive context = sum(w * v for w, v in zip(weights, values)) return context # weighted sum of values # The whole layer, vectorized: # Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V # run in parallel across all tokens, several heads, many layers.

The picture: one query, four keys, one context vector

queryit Thes 0 · w .16 robots 1 · w .43 pickeds 0 · w .16 its 0.5 · w .26 context = Σ w·v = [.31, .85, .31, .52] one block, repeated L times; the lab plays the highlighted step input x LayerNorm attention add x LayerNorm FFN add x
Top: the lab’s default case. s is the scaled score and w the softmax weight; “robot” wins with 0.43, but every token still contributes. Bottom: where that step sits. Each block reads the residual stream x, normalises it, lets every token attend, adds the result back, then runs a per-token feed-forward network and adds again. This is the pre-norm layout most decoders use; a forward pass is that block stacked many times, then a final projection onto the vocabulary.

Worked example: the query “it”, number by number

The lab’s vectors are 4-dimensional, so √d = 2. The query for “it” is [0,1,0,1]; the keys are The [1,0,0,0], robot [0,1,0,1], picked [0,0,1,0], it [0,0,0,1]; each value is 2 in that token’s own slot.

Tokenq · k÷ √de^scoreWeightWeight × value
The001.00000.1571[0.31, 0, 0, 0]
robot212.71830.4269[0, 0.85, 0, 0]
picked001.00000.1571[0, 0, 0.31, 0]
it10.51.64870.2589[0, 0, 0, 0.52]
Sum6.36701.0000[0.31, 0.85, 0.31, 0.52]

Each weight is its exponential divided by 6.3670, and the context vector is the last column summed. Two changes show what the pieces do. Drop the ÷ √d and the scores become 0, 2, 0, 1: the weights sharpen to 0.083, 0.610, 0.083, 0.224. Now give it a decoder’s causal mask and query “robot” instead. It may only see The and itself, so its scores are 0 and 0.5 and its weights 0.378 and 0.622, where the lab’s unmasked version spreads 0.215, 0.355, 0.215, 0.215 over all four. “It” is last in the sentence, so a causal mask changes nothing for it.

Cost and trade-offs: where a forward pass spends its time

Per layer, for n tokens and model width d, counting multiply-adds and assuming the usual feed-forward hidden size of 4d.

CaseTimeSpaceWhy
Q, K, V and output projections4 n d²4 d² weightsEvery token on its own; no token reads another here.
Scores QKᵀ and weights × V2 n² dn² scores per head if storedThe all-pairs part. FlashAttention computes it in tiles so the n² matrix is never held at once.
Feed-forward network8 n d²8 d² weightsPer token again. With the projections, about 12 d² parameters per layer.
Crossover2 n² d = 12 n d² at n = 6d—d = 768: 4,608 tokens. d = 4,096: 24,576. Below that, the matrices cost more than attention.
Decoding one token with a KV cacheO(n d) attention + O(d²) per layer2 × L × n × d cached values32 layers, d = 4,096, 4,096 tokens in fp16: 2 GiB per sequence.

So “attention is quadratic” is true but incomplete. For short and medium contexts the weight matrices dominate compute, and during generation the KV cache dominates memory: it grows with every token of every sequence in the batch, which is what limits how many requests a GPU can serve at once.

Common mistakes

  • Softmax over the wrong axis

    Each query’s weights must sum to 1 across the keys: a row of the score matrix, softmax(scores, axis=-1). Normalise down the columns instead and every shape still checks out while the output is wrong.

  • Masking after the softmax

    A causal mask sets future scores to −∞ before the softmax. Zero them afterwards and the weights no longer sum to 1: for “robot” in the example, 0.215 + 0.355 = 0.570 instead of the correct 0.378 + 0.622.

  • Scaling by the wrong d, or not at all

    Divide by the square root of the per-head key dimension, not the full model width. Leave it out on the lab’s case and the top weight jumps from 0.43 to 0.61; at real dimensions the softmax saturates.

  • A naive softmax that overflows

    exp of a large score overflows to infinity. Subtract the row’s maximum first, as the lab’s own script does; the weights are unchanged because the factor cancels.

  • Forgetting position

    Attention alone ignores order: shuffle the tokens and each one gets the same output as before, just in a new place. “robot picked it” and “it picked robot” would look identical without positional encodings such as RoPE.

Where this shows up

  • Prefill and decode in LLM serving

    A prompt is processed in one parallel forward pass (prefill), which sets time to first token. Each output token then needs its own pass over one new position (decode), reading the KV cache. Serving systems schedule the two phases separately because one is compute-bound and the other memory-bound.

  • Shrinking the KV cache

    Multi-query and grouped-query attention let several query heads share one set of keys and values, cutting the cache by that sharing factor. It is the main reason long contexts fit in memory at all.

  • Encoders, decoders and images

    The lab’s matrix is bidirectional: every token sees all four, as in BERT-style encoders. GPT-style decoders add the causal mask. Vision transformers cut an image into patches and run the same block over them as tokens.

  • Interview angle

    Expect to write single-head attention in NumPy, explain the √d, give the cost (n² d for attention, n d² for the rest) and size a KV cache from layers, width and context length, the way the trade-off table does.

Quick check

1. What are the query, key, and value vectors for a token?

2. How is the attention score between two tokens computed?

3. What is a token’s output (context vector) after attention?

FAQ

What is self-attention in a transformer?

The mechanism that lets each token gather information from every other token. Each token is projected into a query, key, and value; a token’s query is dotted against every key to get relevance scores, which are scaled and softmaxed into weights summing to one, and the token’s new vector is the weighted sum of all values. This learns content-based relationships (like a pronoun attending to its referent) and is repeated across multiple heads and stacked layers to build a transformer.

Why divide the attention scores by the square root of the dimension?

As the query/key dimension grows, the dot-product scores tend to grow in magnitude (more terms summed). Large scores make the softmax extremely peaked, where its gradients become tiny and learning stalls. Dividing by √d rescales the scores to a moderate range so the softmax stays smooth and trainable — a small normalization with an outsized effect on stability, which is why it’s called scaled dot-product attention.

What is multi-head attention and why use several heads?

Multi-head attention runs several attention computations in parallel, each with its own query/key/value projections, then concatenates and projects the results. A single attention pattern captures one kind of relationship at a time; multiple heads let different heads specialize — subject–verb agreement, pronoun resolution, nearby words — at once. Each head works in a lower-dimensional subspace, so total cost is similar to one full attention while greatly enriching what the model can represent.

Why are transformers more parallelizable than RNNs?

An RNN processes a sequence one position at a time, carrying a hidden state forward, so it’s inherently sequential and slow on long sequences. Self-attention lets every token attend to every other in a single step, and all those dot products are independent — so they run simultaneously as big matrix multiplications on a GPU. That parallelism is the main reason transformers replaced RNNs for large-scale training. The trade-off: all-pairs attention is quadratic in sequence length, which is why efficient long-context attention is actively researched.

Keep going

Finished this one? 0 / 61 Labs done
NextRoPE: Position by Rotation Next on the “The transformer, opened up” ladder · rung 3 of 5

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs