Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Handbooks  /  The Transformers Handbook
Handbook~16 min readIntermediateworked arithmetic
Deep Dive

The transformer,
and the attention that runs it.

Every large language model is a stack of the same simple block. Strip away the scale and the mystique, and a transformer is one idea repeated: let every token look at every other token and decide what matters. Here is that idea, built up from tokens to text.

01

The problem it solves

Language is a sequence, and the meaning of a word depends on the words around it — sometimes ones far away. "The animal didn't cross the street because it was too tired" — what does it refer to? To resolve that, a model must connect it back to animal, which could be many words back.

The previous generation of models, RNNs, read a sequence one token at a time, carrying a running summary forward. That has two fatal flaws: it's sequential (you can't process token 100 until you've processed 1–99, so it can't parallelize on modern hardware), and information from far-back tokens gets diluted as it's passed hand-to-hand. The transformer's answer is radical: don't pass information along a chain — let every token directly attend to every other token, all at once.

→ The one-sentence version

A transformer replaces "read left-to-right and remember" with "look at everything at once and weigh what's relevant" — which is both more expressive and fully parallelizable.

02

Tokens, embeddings & position

A model can't consume raw text, so first the text is split into tokens — subword pieces (roughly, common words are one token, rare words split into several). Each token id is mapped to a learned embedding: a vector (say 768 or 4096 numbers) that places the token in a space where similar meanings sit nearby. Now the sequence is a stack of vectors.

But there's a catch. Attention, as we'll see, treats its inputs as an unordered set — it has no built-in sense of "first" or "last". To bidirectional attention, "Dog bites man" and "man bites dog" would look identical. (A causal mask leaks some order, since each token sees only those before it, but only weakly and indirectly.) So we add positional encoding: a signal (fixed sinusoids, or a learned/rotary scheme like RoPE) added to each embedding that tells the model where each token sits. Order is injected as data, not baked into the mechanism.

Text becomes a stack of position-aware vectors
Text
"the cat sat"
→
Tokenize
[the][ cat][ sat]
→
Embed
3 vectors
→
+ Position
who's where
03

Self-attention — the heart

This is the whole idea. Everything else is scaffolding around it.

For each token, the model produces three vectors by multiplying its embedding by three learned weight matrices: a query (Q — "what am I looking for?"), a key (K — "what do I offer?"), and a value (V — "what I'll pass on if chosen"). The metaphor is a lookup: your query is compared against everyone's key to decide whom to listen to.

Concretely, a token's query is dotted with every token's key to produce a relevance score for each. Those scores are scaled and passed through a softmax, turning them into weights that sum to 1 — an attention distribution. The token's new representation is the weighted sum of all the value vectors using those weights. In one line:

Attention(Q,K,V) = softmax( Q·Kᵀ / √d ) · V

Read it as: compare (Q·Kᵀ), normalize (softmax), then gather (·V). The token it can give almost all its weight to animal and pull in its meaning directly — no chain, no dilution, regardless of distance. Every token does this simultaneously, which is why the whole thing runs in parallel.

→ Why Q, K, and V are separate

Splitting into query, key, and value lets a token search by one criterion (Q), advertise itself by another (K), and contribute something else entirely (V) — the flexibility that makes attention so expressive.

04

Multi-head attention

One attention operation forces the model to blend all kinds of relationships into a single set of weights. That's limiting — a word relates to others in many ways at once (grammatical subject, what it modifies, what pronoun refers to it). So transformers run several attention operations in parallel, called heads, each with its own learned Q/K/V projections.

Because each head has different projection matrices, each learns to focus on a different kind of relationship. One head might track the previous word, another might link verbs to their subjects, another might resolve pronouns. The heads' outputs are concatenated and mixed by another matrix, giving the model a rich, multi-perspective view where a single attention would give one flat summary.

One layer, many perspectives

Single attention

  • One set of Q/K/V
  • One relevance pattern
  • All relationships squeezed into one

Multi-head

  • N heads, N projections
  • Each specializes (syntax, coreference, position…)
  • Concatenated → richer representation
05

The transformer block

Attention lets tokens exchange information; the model still needs to process what each token gathered. So each transformer block pairs attention with a small per-token feed-forward network (FFN) — two linear layers with a non-linearity — that transforms each token's vector independently. Attention mixes across tokens; the FFN thinks about each one.

Two more pieces make deep stacks trainable. Residual connections add each sublayer's input to its output, so information (and gradients) can skip layers — you're always refining the representation, never rebuilding it. Layer normalization keeps the numbers in a stable range so training doesn't blow up. The block is just: attention → add & norm → FFN → add & norm.

One block, stacked N times
1Multi-head self-attention — every token gathers from every other.
2Add & norm — residual connection + layer normalization.
3Feed-forward network — transform each token independently.
4Add & norm — again, then hand off to the next block.
A model like GPT is dozens of these identical blocks stacked — same shape, learned weights differ.

That's the entire architecture. "Scaling a transformer" mostly means: bigger embeddings, more heads, more of these blocks, trained on more data. The block never changes.

06

The block's budget — and what changed since 2017

Where do a model's parameters actually live? Count one block's weights at width d: attention needs four d×d projections (Q, K, V, output) = 4d²; the FFN expands to 4d and back = 2·d·4d = 8d². Total ≈ 12d² per layer — and notice the split:

attention : FFN  =  4d² : 8d²  —  two-thirds of a Transformer is the feed-forward network, not attention

That surprises almost everyone: the famous mechanism is the smaller line item. It's also why mixture-of-experts models sparsify the FFN (the big pool) rather than attention, and why "12·L·d²" sizes every model's blocks on a napkin. Add the vocab × d embedding table for the whole model — about 31% of GPT-2 small's 124M, under 0.4% of GPT-3's 175B — and you have the same arithmetic that derives GPT-3's 175B and BERT's 110M on our paper pages.

The 2017 block survives in every modern LLM, but four upgrades became standard. Worth knowing by name — they're the delta between the paper and what you actually run:

ChangeReplacesWhy
RoPE (rotary positions)sinusoidal position additionencodes relative position by rotating Q/K vectors — extrapolates and interpolates to longer contexts far better
GQA (grouped-query attention)one KV head per query head32 query heads share ~8 KV heads → 4× smaller KV cache, nearly free quality-wise; the arithmetic is worked in the LLM Serving handbook
SwiGLU FFNReLU FFNgated activation, better loss per parameter — the FFN got smarter, not just bigger
RMSNorm, pre-normLayerNorm, post-normcheaper, and normalizing before each sublayer trains stably at extreme depth

Want the original attention computed by hand — real Q/K/V matrices, softmax by the numbers, and a runnable 30-line implementation? That lives on our Attention Is All You Need breakdown, worked end to end.

07

From a stack of blocks to text

Modern LLMs are decoder-only: a stack of these blocks with one crucial constraint — causal masking. Each token is only allowed to attend to tokens before it, never ahead. That makes the model a next-token predictor: given everything so far, what comes next?

After the final block, each position's vector is projected over the whole vocabulary and passed through a softmax to get a probability distribution over the next token. To generate, you sample a token from that distribution (temperature and top-p control how adventurous), append it to the sequence, and run the whole thing again for the next token. This loop — predict, sample, append, repeat — is autoregressive generation.

FlavorAttentionUsed for
Encoder-only (BERT)Every token sees all tokens (bidirectional)Understanding — classification, embeddings
Decoder-only (GPT, Llama)Causal — only earlier tokensGeneration — the modern LLM
Encoder–decoder (T5, original)Encoder bidirectional, decoder causal + cross-attendsTranslation, seq-to-seq tasks
→ Where the cost goes

Attention compares every token to every other, so cost grows with the square of the sequence length — the reason long context is expensive and why serving systems cache past keys/values (the KV cache) to avoid recomputing them each step.

08

Why this architecture won

Two properties, together, explain the transformer's dominance. First, parallelism: because every token attends to every other in one operation (not a sequential chain), a whole sequence trains at once, saturating GPUs — which is what made training on internet-scale data feasible. Second, direct long-range connections: any token can reach any other in a single step, so relationships across long distances aren't diluted the way they were in RNNs.

And crucially, it scales predictably — make it bigger and feed it more data and it reliably gets better, with no architectural change. That combination of parallel training, expressive attention, and clean scaling is why the same block underpins GPT, Llama, Claude, and essentially every modern language model. Once you see the block, you've seen the whole family.

By the numbers

Worked example: sizing GPT-2 small from the block formula

The page’s napkin formula is 12·d² per block. Check it against a model whose size is public: GPT-2 small, with width d = 768, 12 blocks, 12 heads, a 50,257-token vocabulary and a 1,024-token context. Biases and norm weights are left out; they are tiny.

StepWhatArithmeticResult
1Attention weights, one block4 × 768²2,359,296
2FFN weights, one block8 × 768²4,718,592
3All 12 blocks12 × 12 × 768²84,934,656
4Token embeddings50,257 × 76838,597,376
5Learned position embeddings1,024 × 768786,432
6Totalsteps 3 + 4 + 5124,318,464 ≈ 124M
7Attention scores, full context1,024² per head × 12 heads × 12 blocks150,994,944 per forward pass
8KV cache per token2 (K and V) × 12 blocks × 76818,432 numbers = 36 KiB at fp16
9KV cache, full context36 KiB × 1,024 tokens36 MiB per sequence

Step 6 lands on GPT-2 small’s published size of about 124M. The output layer adds nothing because GPT-2 reuses (ties) the token-embedding matrix to score the vocabulary. Notice step 4: in a small model the embedding table is almost a third of the weights, so the “two-thirds is FFN” rule applies to the blocks, not the whole model. Steps 7 to 9 are the costs that grow with context rather than with weights. Double the context and step 7 quadruples, while steps 8 and 9 only double. That 36 MiB is per sequence, so a server batching 32 conversations at full context holds over a gigabyte of cache for this tiny model alone.

Trade-offs

Cost and trade-offs: where the compute and memory go

Here n is the sequence length, d the model width and L the number of blocks. Costs are per block unless stated.

CaseTimeSpaceWhy
Attention scores (training, prefill)O(n²·d)O(n²) per head if the score matrix is storedEvery token is compared with every other. This is the term that makes long context expensive.
Projections + FFNO(n·d²)O(d²) weightsLinear in n. Counting 2 FLOPs per multiply-add, weights cost 24·d² per token and attention 4·n·d, so they break even at n = 6d: 4,608 tokens for GPT-2’s d = 768.
Decode one token, with KV cacheO(n·d + d²)Cache grows O(L·d) per tokenThe new token’s query reads every cached key and value. Speed is bounded by memory bandwidth, not arithmetic.
Decode without a KV cacheO(n²·d + n·d²) every stepNo cacheEach new token reprocesses the whole prefix. Nobody serves this way.
FlashAttentionSame O(n²·d)O(n) extraComputes exact attention in tiles that fit on-chip, so the n×n matrix is never written to GPU memory. Faster in practice, same answer.
GQA (page section 6)Same computeKV cache ÷ (query heads ÷ KV heads)32 query heads on 8 KV heads cuts the cache 4×.
RNN, for contrastO(n·d²) totalO(d) stateCheaper on paper, but n steps that must run one after another, so it cannot use a GPU’s parallelism during training.

The short version: weights dominate at short context, the n² term dominates at long context, and during generation the KV cache, not the arithmetic, sets how many users one GPU can serve.

Watch for

Common mistakes

  • Dropping the √d

    The dot product of two d-dimensional vectors with unit-variance entries has variance d. Unscaled, scores at d = 64 are about 8× too spread out, softmax saturates to nearly one-hot, and gradients through it vanish. The /√d in the formula keeps the scores in softmax’s useful range.

  • Thinking more heads means more parameters

    Heads split the width; they do not add to it. 12 heads of 64 dimensions use the same four d×d matrices as one head of 768. Attention stays at 4d² whatever the head count. Only the number of separate attention patterns changes.

  • Forgetting the causal mask in training

    Train a decoder without it and each position can see the token it is meant to predict. Loss drops to almost nothing, then generation is gibberish, because at inference the future is not there to copy. A suspiciously perfect first epoch usually means this bug.

  • Letting padding into attention

    Batches pad short sequences to one length. Without a padding mask, real tokens attend to pad tokens, and the same input gives different outputs depending on what it was batched with. Mask pad positions to −∞ before the softmax.

  • Reading attention weights as an explanation

    A head putting weight on animal is suggestive, not proof of why the model answered as it did. Values are mixed again by later layers, residual paths and the FFN. Research has shown that quite different attention patterns can produce the same prediction.

In the wild

Where this shows up in production and interviews

  • LLM serving

    Prefill (reading the prompt) is the O(n²·d), compute-bound phase. Decode (one token at a time) is memory-bound and lives or dies by the KV cache from step 9. Paged KV caches and continuous batching exist to pack more sequences into that memory. The LLM Serving handbook works the numbers.

  • Long context

    The n² term is why long context is priced and engineered separately. FlashAttention removes the n×n memory cost without approximating. Sliding-window attention, used by Mistral 7B, caps how far back each layer looks, so per-token cost stops growing with n.

  • Beyond text

    The block does not care what a token is. Vision Transformers cut an image into 16×16-pixel patches and treat each one as a token. Whisper runs an encoder–decoder transformer over audio spectrograms. Embedding models are encoder-only stacks with one vector pooled out at the end.

  • The interview version

    Expect to derive 12·L·d² and check it on a known model as above, explain why attention is O(n²), size a KV cache, say why the √d is there and what the causal mask prevents. Getting the arithmetic right counts for more than naming the latest variant.

Self-check

Check yourself

A model has d = 4096 and 32 heads. How wide is each head, and does moving to 64 heads change the attention parameter count?

Each head is 4096 ÷ 32 = 128 wide. At 64 heads each is 64 wide, but the Q, K, V and output matrices are still 4096×4096, so attention stays at 4d². Head count changes how the width is divided, not how many weights there are.

You double the context from 4k to 8k tokens. What happens to the attention-score work, the FFN work and the KV cache?

Score work is O(n²), so it grows 4×. FFN and projection work is O(n), so it doubles. The KV cache holds one K and V per token per block, so it doubles too. Long context gets expensive because one part of the cost grows faster than everything else.

Which adds more parameters to a 12·L·d² model: doubling the width or doubling the depth?

Doubling d multiplies the block weights by 4, because d is squared. Doubling L multiplies them by 2. That is why parameter budgets are usually spent on width and depth together rather than on one alone.

Your decoder’s training loss is near zero after one epoch, but sampling produces nonsense. What do you check first?

The causal mask. Without it each position can attend to the next token, which is the training target, so the model learns to copy instead of predict. At generation the future is not there, and the copying strategy collapses.

Frequently asked

Quick answers

What is a transformer?

A neural network architecture built around self-attention, letting every token in a sequence directly attend to every other token. It processes sequences in parallel and captures long-range dependencies, which is why it underlies modern LLMs.

What is self-attention?

Each token produces a query, key, and value; it scores its query against every token's key, softmaxes those into weights, and takes a weighted sum of the values — gathering information from the most relevant tokens regardless of distance.

Why multiple heads?

Each head has its own Q/K/V projections and can specialize in a different relationship (syntax, coreference, position). Concatenating heads gives a richer, multi-perspective representation than a single attention.

How does it generate text?

A decoder-only transformer predicts a probability distribution over the next token, samples one, appends it, and repeats — autoregressive generation, with causal masking so each position only sees earlier tokens.

Finished this one? 0 / 209 Handbooks done
NextThe Embeddings Handbook Next on the “How models actually work” ladder · rung 3 of 11

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks