Attention Is All You Need,
explained.

Rung 1 reads a sentence one token at a time, and everything about it follows from that: information from position 1 reaches position 40 by passing through 39 intermediate states, and position 40 cannot be computed until position 39 has been. The first property is a quality problem, the second is a speed problem, and this rung removes both by removing recurrence.

The change: every position attends directly to every other position in a single operation. Each token emits a query, a key and a value; a token's new representation is a weighted average of all the values, weighted by how well its query matches each key. The path length between any two positions becomes one step instead of n, which is the quality half. And because no position waits for another, an entire sequence is processed in parallel during training, which is the speed half and was the more immediately decisive of the two — multi-head attention runs several such comparisons in parallel subspaces so one layer can attend to different kinds of relation at once.

Two consequences are worth carrying up the ladder because later rungs are about them. First, attention is permutation-invariant — with no recurrence there is nothing that knows the order of the tokens — so position has to be added in explicitly, which this paper does with fixed sinusoidal vectors added to the embeddings, noting that learned position embeddings perform about as well. Rung 3 is an argument that adding is the wrong operation. Second, comparing every position with every other is quadratic in sequence length; the paper names this as the cost and mentions restricting attention to a neighbourhood as a remedy for long sequences, which is the entire context ladder in one sentence of future work.

Before this paper, AI read sentences the way you'd play a game of telephone. One word passes its meaning to the next, which passes it to the next, in a strict single-file line — a recurrent neural network (RNN). By the time the model reaches word thirty, the signal from word one has degraded or vanished entirely.

Worse, because each word has to wait for the one before it, this process can't be parallelized within a single sequence. Throwing more GPUs at an RNN doesn't make one sentence process faster — it's fundamentally, stubbornly sequential.

The authors asked a deceptively simple question: what if no word had to wait in line at all? What if every word could look directly at every other word, all at once, no matter how far apart they are?

It's like being at a party where you can tune into any conversation in the room instantly, and turn the volume up or down on each one depending on how relevant it is to you right now. That mechanism is self-attention — and the paper's entire bet was that you don't need recurrence, and you don't need convolutions. Attention alone is enough.

So how does one word actually "look at" another? With three simple ideas: a Query, a Key, and a Value.

Think of it like searching a library. Your Query is what you're looking for. Every word carries a Key, like a label on a book's spine — you compare your query against every key to get a relevance score. Then you pull from each word's Value, its actual content, weighted by how relevant it was. Highly relevant words you read closely; irrelevant ones you barely glance at. Do this for every word against every other word, and each word ends up with a representation soaked in exactly the context that matters.

The paper's core formula: Attention(Q,K,V) = softmax( QK^^T / √d_k ) V.

Comparing a query and key with a dot product can produce very large numbers, and large numbers push the softmax into an all-or-nothing collapse — terrible for learning. So the paper divides every score by the square root of the key dimension before the softmax: one line of math that, the authors argue from the variance of the dot products, keeps the softmax out of its flat, near-zero-gradient regions. (The paper argues this rather than ablating it.)

Three tokens, dimension d_k = 2 — small enough that every number fits in your head. Say the projections have already produced:

Follow token 2 (query [0, 2]) through the formula, step by step:

That output row is token 2's new representation: ~45% of it pulled from token 2's own value, ~45% from token 3, and a light 11% glance at token 1. "Attention" is exactly this — a learned, differentiable weighted average, recomputed for every token against every other.

Instead of computing attention once, the Transformer runs it eight times in parallel, each with its own learned projections. One head might specialize in grammar, another in what a pronoun refers to, another in position — like sending eight detectives to examine the same sentence, each hunting for a different kind of clue, then pooling their notes into one conclusion.

The dimensions make it cheap. The model width is d_model = 512, but each head works in a slice: its W_Q, W_K, W_V matrices project 512 → 64, so 8 heads × 64 dims = 512 total — the same arithmetic budget as one full-width head, just split into eight independent viewpoints. The eight 64-dim outputs are concatenated back to 512 and mixed once more by an output matrix W_O:

The paper's ablations back the design up: a single wide head is 0.9 BLEU worse than 8 slim ones — but 32 heads is worse again. The diversity of viewpoints matters more than the width of any one.

Stack that idea into a full architecture and you get the Transformer: an encoder that reads the input, and a decoder that writes the output, each built from six identical layers. Every layer does attention, then a small feed-forward network, wrapped in a residual connection and layer normalization so information flows cleanly even six layers deep.

The decoder gets one extra rule: it's masked so it can't peek at future tokens while generating, or it could just cheat by reading the answer it's supposed to produce.

If every word looks at every other word all at once, how does the model know what order they're in? Attention on its own has no sense of sequence. The fix is almost poetic: give every word a unique positional fingerprint made of overlapping sine and cosine waves at different frequencies — like a wristwatch reading of exactly where it sits in the sentence. Add that fingerprint in before anything else happens, and order is baked in without a single recurrent step.

Each dimension pair (2i, 2i+1) is a sine/cosine at its own wavelength, from 2π up to 10000·2π — fast-ticking dimensions distinguish neighbors, slow ones distinguish distant regions, together forming a unique code for every position. The sinusoid choice has a bonus property: PE(pos+k) is a linear function of PE(pos), so "k steps apart" is easy for attention to express. The paper also tried fully learned position embeddings — nearly identical scores — and kept the sinusoids because they might extrapolate to sequence lengths longer than any seen in training — a hypothesis the paper states but does not test.

A recurrent network needs one sequential step per token, so a hundred-word sentence takes a hundred unavoidable steps. Self-attention connects any two words in exactly one step, no matter the distance — the whole sequence gets processed at once, like a room full of people all talking simultaneously instead of one by one.

Vaswani et al. built the Transformer as an encoder-decoder for translation: the same outer shape as seq2seq, with every recurrent part replaced. The base model has d_model = 512, 8 heads of 64 dimensions, 6 encoder and 6 decoder layers and 65M parameters. The big model widens to 1,024 with 16 heads, for 213M.

1. Tokenize into subwords. English-German used byte-pair encoding with a shared source-target vocabulary of about 37,000 tokens; English-French used 32,000 word-pieces. One embedding matrix is shared by the encoder input, the decoder input and the final pre-softmax projection. Embeddings are multiplied by √d_model, then the sinusoidal position vectors are added.
2. Encoder layer: self-attention, then a feed-forward network. Queries, keys and values all come from the previous encoder layer, so every source position attends to every source position. The second sub-layer is a position-wise feed-forward network, FFN(x) = max(0, xW₁ + b₁)W₂ + b₂, which expands 512 dimensions to 2,048 and back with a ReLU in between. It applies the same weights at every position (two 1×1 convolutions, in effect) and different weights in each layer.
3. Wrap every sub-layer the same way. A sub-layer's output goes through dropout, is added to the sub-layer's input, and the sum is layer-normalized: LayerNorm(x + Sublayer(x)). Normalization comes after the residual addition, the arrangement now called post-norm.
4. Decoder layer: three sub-layers. First, masked self-attention over the target. Scores for every position after the current one are set to −∞ before the softmax, and the target is shifted right by one, so the prediction at position i can only use tokens before i. Second, encoder-decoder attention, also called cross-attention. Its queries come from the decoder and its keys and values come from the final encoder layer's output, so every decoder position can read the whole source in one step. Third, a feed-forward network of the same form. All three get the residual-plus-LayerNorm wrapper.
5. Predict. A linear layer, tied to the embeddings, and a softmax turn the top decoder state into next-token probabilities.
6. Batch by length and optimize. Sentence pairs of similar length were grouped so that each batch held about 25,000 source and 25,000 target tokens. Adam ran with β₁ = 0.9, β₂ = 0.98 and ε = 10⁻⁹. The learning rate rises linearly for 4,000 warmup steps, then decays with the inverse square root of the step number.
7. Regularize twice. Dropout of 0.1 (base) is applied to every sub-layer output and to the sums of embeddings and positional encodings. Label smoothing of 0.1 moves a tenth of the target probability mass off the correct token, which the paper notes hurts perplexity but improves accuracy and BLEU.
8. Train, then decode. On one machine with 8 NVIDIA P100 GPUs, the base model ran 100,000 steps at about 0.4 seconds each (12 hours) and the big model ran 300,000 steps at about 1.0 second each (3.5 days). For evaluation, the last 5 checkpoints (base) or last 20 (big) were averaged. Translations came from beam search with beam size 4 and length penalty α = 0.6, with output capped at input length plus 50.

What each layer type costs

The paper argues for self-attention by comparing three properties per layer, for sequence length n, width d, convolution kernel k and neighbourhood size r:

Self-attention does less work than recurrence whenever n is smaller than d, which held for sentence-level translation at d = 512. The restricted row is the remedy the paper proposes for very long inputs.

• qi, ki — components of a query and a key, taken as independent with mean 0 and variance 1. Each product then has variance 1, and d_k of them sum to variance d_k.
• √dk — the divisor that brings scores back to unit variance, so the softmax does not saturate into near-zero gradients as heads get wider. At d_k = 64, unscaled scores would have a standard deviation of 8.
• dmodel^−0.5 — ties the size of every step to the model width; for 512 it is about 0.044.
• step · warmup^−1.5 — the linear ramp, the smaller of the two terms until step equals warmup = 4,000.
• step^−0.5 — the inverse-square-root decay that takes over after warmup.

At the extremes: the two terms meet at step 4,000, where the rate peaks at (512 · 4,000)^^−0.5 ≈ 7.0 × 10⁻⁴; by step 100,000 it has fallen to about 1.4 × 10⁻⁴. Setting warmup to 1 removes the ramp and starts training at the largest rate; later analyses such as Xiong et al. (2020) tied the need for warmup to the post-norm placement. For the scores, a small d_k makes the scaling nearly irrelevant, which matches the earlier comparisons the paper cites: dot-product and additive attention perform alike at small d_k, and additive wins over unscaled dot products at large d_k.

Training data was WMT 2014: about 4.5 million English-German sentence pairs and 36 million English-French pairs. Results are on newstest2014, and ablations use newstest2013 as a development set. Everything ran on one machine with 8 P100 GPUs, and the paper compares training cost in estimated floating-point operations.

The big model beat the best previous English-German result, ensembles included, by more than 2 BLEU, and even the base model beat every earlier model and ensemble at a fraction of their training cost. English-French was closer: the base model's 38.1 trailed the strongest single models, and only the big model, trained for 3.5 days, set the single-model record. That model used dropout 0.1 instead of the 0.3 used for big English-German.

Ablations on the development set (base: 25.8 BLEU, 65M parameters; big: 26.4 BLEU, 213M) isolate the design choices. One attention head was 0.9 BLEU worse than eight, and 32 heads was also worse. Shrinking the key size d_k hurt quality, which the authors read as a sign that judging compatibility is not easy. Removing dropout cost more than a point. Turning label smoothing off improved perplexity but lowered BLEU. Learned position embeddings matched sinusoids almost exactly, 25.7 against 25.8.

A second task tested generality: English constituency parsing on the Wall Street Journal part of the Penn Treebank, about 40,000 training sentences, with a 4-layer, 1,024-wide model and little task-specific tuning. It reached 91.3 F1 on WSJ data alone and 92.7 F1 semi-supervised, ahead of every earlier model the authors compared against except the Recurrent Neural Network Grammar.

The quadratic cost is stated rather than discovered later. So is the empirical basis for several choices: the number of heads, the scaled-dot-product form and the feed-forward width are established by ablation rather than derived, and the paper presents them that way. It is a machine translation paper reporting machine translation numbers, and it does not claim the architecture is a general substrate for language — that claim was made by what happened next.

Two of its specifics lost. The sinusoidal position encoding was displaced, first by learned absolute embeddings and then by the relative and rotary schemes of rung 3. And the encoder-decoder shape lost to decoder-only stacks for generative models, for reasons of efficiency and simplicity rather than quality — one stack, one objective, and no cross-attention to maintain.

How it held up: it is the most load-bearing paper on this site. Essentially every language, vision-language and audio model in the catalog is this architecture with the positions changed, the normaliser moved (pre-norm), the feed-forward layer made sparse, or the attention kernel made memory-aware. The quadratic cost it flagged in future work became the defining engineering constraint of the decade that followed, and the several billion dollars spent working around it are the best possible evidence the authors identified the right weakness.

That general-purpose idea is why, years later, this paper still matters. GPT, BERT, Claude, Gemini — strip away the branding, and the Transformer is the engine underneath every one of them. One architecture, one core idea — let every element look directly at every other element — quietly became the foundation of modern AI.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The architecture itself: what it replaced, how position is encoded once recurrence is gone, and what it turns out to have learned inside.

1. 1Seq2Seq2014
2. 2Attention Is All You Need2017
3. 3RoPE2021
4. 4Concept Circuits2026

All 110 papers, in 19 ladders →