Attention Is All You Need,
explained.

Before this paper, AI read sentences the way you'd play a game of telephone. One word passes its meaning to the next, which passes it to the next, in a strict single-file line — a recurrent neural network (RNN). By the time the model reaches word thirty, the signal from word one has degraded or vanished entirely.

Worse, because each word has to wait for the one before it, this process can't be parallelized within a single sequence. Throwing more GPUs at an RNN doesn't make one sentence process faster — it's fundamentally, stubbornly sequential.

The authors asked a deceptively simple question: what if no word had to wait in line at all? What if every word could look directly at every other word, all at once, no matter how far apart they are?

It's like being at a party where you can tune into any conversation in the room instantly, and turn the volume up or down on each one depending on how relevant it is to you right now. That mechanism is self-attention — and the paper's entire bet was that you don't need recurrence, and you don't need convolutions. Attention alone is enough.

So how does one word actually "look at" another? With three simple ideas: a Query, a Key, and a Value.

Think of it like searching a library. Your Query is what you're looking for. Every word carries a Key, like a label on a book's spine — you compare your query against every key to get a relevance score. Then you pull from each word's Value, its actual content, weighted by how relevant it was. Highly relevant words you read closely; irrelevant ones you barely glance at. Do this for every word against every other word, and each word ends up with a representation soaked in exactly the context that matters.

The paper's core formula: Attention(Q,K,V) = softmax( QK^T / √d_k ) V.

Comparing a query and key with a dot product can produce very large numbers, and large numbers push the softmax into an all-or-nothing collapse — terrible for learning. So the paper divides every score by the square root of the key dimension before the softmax: one line of math that keeps training stable.

Three tokens, dimension d_k = 2 — small enough that every number fits in your head. Say the projections have already produced:

Follow token 2 (query [0, 2]) through the formula, step by step:

That output row is token 2's new representation: ~45% of it pulled from token 2's own value, ~45% from token 3, and a light 11% glance at token 1. "Attention" is exactly this — a learned, differentiable weighted average, recomputed for every token against every other.

Instead of computing attention once, the Transformer runs it eight times in parallel, each with its own learned projections. One head might specialize in grammar, another in what a pronoun refers to, another in position — like sending eight detectives to examine the same sentence, each hunting for a different kind of clue, then pooling their notes into one conclusion.

The dimensions make it cheap. The model width is d_model = 512, but each head works in a slice: its W_Q, W_K, W_V matrices project 512 → 64, so 8 heads × 64 dims = 512 total — the same arithmetic budget as one full-width head, just split into eight independent viewpoints. The eight 64-dim outputs are concatenated back to 512 and mixed once more by an output matrix W_O:

The paper's ablations back the design up: a single wide head is 0.9 BLEU worse than 8 slim ones — but 32 heads is worse again. The diversity of viewpoints matters more than the width of any one.

Stack that idea into a full architecture and you get the Transformer: an encoder that reads the input, and a decoder that writes the output, each built from six identical layers. Every layer does attention, then a small feed-forward network, wrapped in a residual connection and layer normalization so information flows cleanly even six layers deep.

The decoder gets one extra rule: it's masked so it can't peek at future tokens while generating, or it could just cheat by reading the answer it's supposed to produce.

If every word looks at every other word all at once, how does the model know what order they're in? Attention on its own has no sense of sequence. The fix is almost poetic: give every word a unique positional fingerprint made of overlapping sine and cosine waves at different frequencies — like a wristwatch reading of exactly where it sits in the sentence. Add that fingerprint in before anything else happens, and order is baked in without a single recurrent step.

Each dimension pair (2i, 2i+1) is a sine/cosine at its own wavelength, from 2π up to 10000·2π — fast-ticking dimensions distinguish neighbors, slow ones distinguish distant regions, together forming a unique code for every position. The sinusoid choice has a bonus property: PE(pos+k) is a linear function of PE(pos), so "k steps apart" is easy for attention to express. The paper also tried fully learned position embeddings — nearly identical scores — and kept the sinusoids because they extrapolate to lengths never seen in training.

A recurrent network needs one sequential step per token, so a hundred-word sentence takes a hundred unavoidable steps. Self-attention connects any two words in exactly one step, no matter the distance — the whole sequence gets processed at once, like a room full of people all talking simultaneously instead of one by one.

On WMT 2014 English-to-German translation, the Transformer (big) scored 28.4 BLEU, beating every previous best — including model ensembles — by more than two points, while training in a fraction of the compute. On English-to-French it set a new single-model record of 41.8 BLEU, trained in 3.5 days on eight GPUs.

The ablation table (Table 3 in the paper) is quietly the most instructive part: one attention head costs 0.9 BLEU versus eight; dropping the √d_k scaling hurts; and bigger models are reliably better — the base model has 65M parameters, the big one 213M. Learned positional embeddings scored the same as sinusoids, so the sinusoids won on extrapolation, not accuracy.

Skeptics might say it only works for translation — so the authors threw it at a completely different task, English constituency parsing, with almost no tuning. A 4-layer Transformer hit 91.3 F1 trained on the WSJ corpus alone and 92.7 F1 semi-supervised — beating nearly every specialized parser built for that job. The architecture wasn't a translation trick; it was a general-purpose idea.

That general-purpose idea is why, years later, this paper still matters. GPT, BERT, Claude, Gemini — strip away the branding, and the Transformer is the engine underneath every one of them. One architecture, one core idea — let every element look directly at every other element — quietly became the foundation of modern AI.