Attention Is All You Need,
explained.

Rung 1 reads a sentence one token at a time, and everything about it follows from that: information from position 1 reaches position 40 by passing through 39 intermediate states, and position 40 cannot be computed until position 39 has been. The first property is a quality problem, the second is a speed problem, and this rung removes both by removing recurrence.

The change: every position attends directly to every other position in a single operation. Each token emits a query, a key and a value; a token's new representation is a weighted average of all the values, weighted by how well its query matches each key. The path length between any two positions becomes one step instead of n, which is the quality half. And because no position waits for another, an entire sequence is processed in parallel during training, which is the speed half and was the more immediately decisive of the two — multi-head attention runs several such comparisons in parallel subspaces so one layer can attend to different kinds of relation at once.

Two consequences are worth carrying up the ladder because later rungs are about them. First, attention is permutation-invariant — with no recurrence there is nothing that knows the order of the tokens — so position has to be added in explicitly, which this paper does with fixed sinusoidal vectors added to the embeddings, noting that learned position embeddings perform about as well. Rung 3 is an argument that adding is the wrong operation. Second, comparing every position with every other is quadratic in sequence length; the paper names this as the cost and mentions restricting attention to a neighbourhood as a remedy for long sequences, which is the entire context ladder in one sentence of future work.

Before this paper, AI read sentences the way you'd play a game of telephone. One word passes its meaning to the next, which passes it to the next, in a strict single-file line — a recurrent neural network (RNN). By the time the model reaches word thirty, the signal from word one has degraded or vanished entirely.

Worse, because each word has to wait for the one before it, this process can't be parallelized within a single sequence. Throwing more GPUs at an RNN doesn't make one sentence process faster — it's fundamentally, stubbornly sequential.

The authors asked a deceptively simple question: what if no word had to wait in line at all? What if every word could look directly at every other word, all at once, no matter how far apart they are?

It's like being at a party where you can tune into any conversation in the room instantly, and turn the volume up or down on each one depending on how relevant it is to you right now. That mechanism is self-attention — and the paper's entire bet was that you don't need recurrence, and you don't need convolutions. Attention alone is enough.

So how does one word actually "look at" another? With three simple ideas: a Query, a Key, and a Value.

Think of it like searching a library. Your Query is what you're looking for. Every word carries a Key, like a label on a book's spine — you compare your query against every key to get a relevance score. Then you pull from each word's Value, its actual content, weighted by how relevant it was. Highly relevant words you read closely; irrelevant ones you barely glance at. Do this for every word against every other word, and each word ends up with a representation soaked in exactly the context that matters.

The paper's core formula: Attention(Q,K,V) = softmax( QK^T / √d_k ) V.

Comparing a query and key with a dot product can produce very large numbers, and large numbers push the softmax into an all-or-nothing collapse — terrible for learning. So the paper divides every score by the square root of the key dimension before the softmax: one line of math that keeps training stable.

Three tokens, dimension d_k = 2 — small enough that every number fits in your head. Say the projections have already produced:

Follow token 2 (query [0, 2]) through the formula, step by step:

That output row is token 2's new representation: ~45% of it pulled from token 2's own value, ~45% from token 3, and a light 11% glance at token 1. "Attention" is exactly this — a learned, differentiable weighted average, recomputed for every token against every other.

Instead of computing attention once, the Transformer runs it eight times in parallel, each with its own learned projections. One head might specialize in grammar, another in what a pronoun refers to, another in position — like sending eight detectives to examine the same sentence, each hunting for a different kind of clue, then pooling their notes into one conclusion.

The dimensions make it cheap. The model width is d_model = 512, but each head works in a slice: its W_Q, W_K, W_V matrices project 512 → 64, so 8 heads × 64 dims = 512 total — the same arithmetic budget as one full-width head, just split into eight independent viewpoints. The eight 64-dim outputs are concatenated back to 512 and mixed once more by an output matrix W_O:

The paper's ablations back the design up: a single wide head is 0.9 BLEU worse than 8 slim ones — but 32 heads is worse again. The diversity of viewpoints matters more than the width of any one.

Stack that idea into a full architecture and you get the Transformer: an encoder that reads the input, and a decoder that writes the output, each built from six identical layers. Every layer does attention, then a small feed-forward network, wrapped in a residual connection and layer normalization so information flows cleanly even six layers deep.

The decoder gets one extra rule: it's masked so it can't peek at future tokens while generating, or it could just cheat by reading the answer it's supposed to produce.

If every word looks at every other word all at once, how does the model know what order they're in? Attention on its own has no sense of sequence. The fix is almost poetic: give every word a unique positional fingerprint made of overlapping sine and cosine waves at different frequencies — like a wristwatch reading of exactly where it sits in the sentence. Add that fingerprint in before anything else happens, and order is baked in without a single recurrent step.

Each dimension pair (2i, 2i+1) is a sine/cosine at its own wavelength, from 2π up to 10000·2π — fast-ticking dimensions distinguish neighbors, slow ones distinguish distant regions, together forming a unique code for every position. The sinusoid choice has a bonus property: PE(pos+k) is a linear function of PE(pos), so "k steps apart" is easy for attention to express. The paper also tried fully learned position embeddings — nearly identical scores — and kept the sinusoids because they extrapolate to lengths never seen in training.

A recurrent network needs one sequential step per token, so a hundred-word sentence takes a hundred unavoidable steps. Self-attention connects any two words in exactly one step, no matter the distance — the whole sequence gets processed at once, like a room full of people all talking simultaneously instead of one by one.

The quadratic cost is stated rather than discovered later. So is the empirical basis for several choices: the number of heads, the scaled-dot-product form and the feed-forward width are established by ablation rather than derived, and the paper presents them that way. It is a machine translation paper reporting machine translation numbers, and it does not claim the architecture is a general substrate for language — that claim was made by what happened next.

Two of its specifics lost. The sinusoidal position encoding was displaced, first by learned absolute embeddings and then by the relative and rotary schemes of rung 3. And the encoder-decoder shape lost to decoder-only stacks for generative models, for reasons of efficiency and simplicity rather than quality — one stack, one objective, and no cross-attention to maintain.

How it held up: it is the most load-bearing paper on this site. Essentially every language, vision-language and audio model in the catalog is this architecture with the positions changed, the normaliser moved (pre-norm), the feed-forward layer made sparse, or the attention kernel made memory-aware. The quadratic cost it flagged in future work became the defining engineering constraint of the decade that followed, and the several billion dollars spent working around it are the best possible evidence the authors identified the right weakness.

On WMT 2014 English-to-German translation, the Transformer (big) scored 28.4 BLEU, beating every previous best — including model ensembles — by more than two points, while training in a fraction of the compute. On English-to-French it set a new single-model record of 41.8 BLEU, trained in 3.5 days on eight GPUs.

The ablation table (Table 3 in the paper) is quietly the most instructive part: one attention head costs 0.9 BLEU versus eight; dropping the √d_k scaling hurts; and bigger models are reliably better — the base model has 65M parameters, the big one 213M. Learned positional embeddings scored the same as sinusoids, so the sinusoids won on extrapolation, not accuracy.

Skeptics might say it only works for translation — so the authors threw it at a completely different task, English constituency parsing, with almost no tuning. A 4-layer Transformer hit 91.3 F1 trained on the WSJ corpus alone and 92.7 F1 semi-supervised — beating nearly every specialized parser built for that job. The architecture wasn't a translation trick; it was a general-purpose idea.

That general-purpose idea is why, years later, this paper still matters. GPT, BERT, Claude, Gemini — strip away the branding, and the Transformer is the engine underneath every one of them. One architecture, one core idea — let every element look directly at every other element — quietly became the foundation of modern AI.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The architecture itself: what it replaced, how position is encoded once recurrence is gone, and what it turns out to have learned inside.

1. 1Seq2Seq2014
2. 2Attention Is All You Need2017
3. 3RoPE2021
4. 4Concept Circuits2026

All 110 papers, in 19 ladders →