Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
The transformer,
and the attention that runs it.
Every large language model is a stack of the same simple block. Strip away the scale and the mystique, and a transformer is one idea repeated: let every token look at every other token and decide what matters. Here is that idea, built up from tokens to text.
The problem it solves
Language is a sequence, and the meaning of a word depends on the words around it — sometimes ones far away. "The animal didn't cross the street because it was too tired" — what does it refer to? To resolve that, a model must connect it back to animal, which could be many words back.
The previous generation of models, RNNs, read a sequence one token at a time, carrying a running summary forward. That has two fatal flaws: it's sequential (you can't process token 100 until you've processed 1–99, so it can't parallelize on modern hardware), and information from far-back tokens gets diluted as it's passed hand-to-hand. The transformer's answer is radical: don't pass information along a chain — let every token directly attend to every other token, all at once.
A transformer replaces "read left-to-right and remember" with "look at everything at once and weigh what's relevant" — which is both more expressive and fully parallelizable.
Tokens, embeddings & position
A model can't consume raw text, so first the text is split into tokens — subword pieces (roughly, common words are one token, rare words split into several). Each token id is mapped to a learned embedding: a vector (say 768 or 4096 numbers) that places the token in a space where similar meanings sit nearby. Now the sequence is a stack of vectors.
But there's a catch. Attention, as we'll see, treats its inputs as an unordered set — it has no built-in sense of "first" or "last". To bidirectional attention, "Dog bites man" and "man bites dog" would look identical. (A causal mask leaks some order, since each token sees only those before it, but only weakly and indirectly.) So we add positional encoding: a signal (fixed sinusoids, or a learned/rotary scheme like RoPE) added to each embedding that tells the model where each token sits. Order is injected as data, not baked into the mechanism.
Self-attention — the heart
This is the whole idea. Everything else is scaffolding around it.
For each token, the model produces three vectors by multiplying its embedding by three learned weight matrices: a query (Q — "what am I looking for?"), a key (K — "what do I offer?"), and a value (V — "what I'll pass on if chosen"). The metaphor is a lookup: your query is compared against everyone's key to decide whom to listen to.
Concretely, a token's query is dotted with every token's key to produce a relevance score for each. Those scores are scaled and passed through a softmax, turning them into weights that sum to 1 — an attention distribution. The token's new representation is the weighted sum of all the value vectors using those weights. In one line:
Attention(Q,K,V) = softmax( Q·Kᵀ / √d ) · V
Read it as: compare (Q·Kᵀ), normalize (softmax), then gather (·V). The token it can give almost all its weight to animal and pull in its meaning directly — no chain, no dilution, regardless of distance. Every token does this simultaneously, which is why the whole thing runs in parallel.
Splitting into query, key, and value lets a token search by one criterion (Q), advertise itself by another (K), and contribute something else entirely (V) — the flexibility that makes attention so expressive.
Multi-head attention
One attention operation forces the model to blend all kinds of relationships into a single set of weights. That's limiting — a word relates to others in many ways at once (grammatical subject, what it modifies, what pronoun refers to it). So transformers run several attention operations in parallel, called heads, each with its own learned Q/K/V projections.
Because each head has different projection matrices, each learns to focus on a different kind of relationship. One head might track the previous word, another might link verbs to their subjects, another might resolve pronouns. The heads' outputs are concatenated and mixed by another matrix, giving the model a rich, multi-perspective view where a single attention would give one flat summary.
Single attention
- One set of Q/K/V
- One relevance pattern
- All relationships squeezed into one
Multi-head
- N heads, N projections
- Each specializes (syntax, coreference, position…)
- Concatenated → richer representation
The transformer block
Attention lets tokens exchange information; the model still needs to process what each token gathered. So each transformer block pairs attention with a small per-token feed-forward network (FFN) — two linear layers with a non-linearity — that transforms each token's vector independently. Attention mixes across tokens; the FFN thinks about each one.
Two more pieces make deep stacks trainable. Residual connections add each sublayer's input to its output, so information (and gradients) can skip layers — you're always refining the representation, never rebuilding it. Layer normalization keeps the numbers in a stable range so training doesn't blow up. The block is just: attention → add & norm → FFN → add & norm.
That's the entire architecture. "Scaling a transformer" mostly means: bigger embeddings, more heads, more of these blocks, trained on more data. The block never changes.
The block's budget — and what changed since 2017
Where do a model's parameters actually live? Count one block's weights at width d: attention needs four d×d projections (Q, K, V, output) = 4d²; the FFN expands to 4d and back = 2·d·4d = 8d². Total ≈ 12d² per layer — and notice the split:
That surprises almost everyone: the famous mechanism is the smaller line item. It's also why mixture-of-experts models sparsify the FFN (the big pool) rather than attention, and why "12·L·d²" sizes every model's blocks on a napkin. Add the vocab × d embedding table for the whole model — about 31% of GPT-2 small's 124M, under 0.4% of GPT-3's 175B — and you have the same arithmetic that derives GPT-3's 175B and BERT's 110M on our paper pages.
The 2017 block survives in every modern LLM, but four upgrades became standard. Worth knowing by name — they're the delta between the paper and what you actually run:
| Change | Replaces | Why |
|---|---|---|
| RoPE (rotary positions) | sinusoidal position addition | encodes relative position by rotating Q/K vectors — extrapolates and interpolates to longer contexts far better |
| GQA (grouped-query attention) | one KV head per query head | 32 query heads share ~8 KV heads → 4× smaller KV cache, nearly free quality-wise; the arithmetic is worked in the LLM Serving handbook |
| SwiGLU FFN | ReLU FFN | gated activation, better loss per parameter — the FFN got smarter, not just bigger |
| RMSNorm, pre-norm | LayerNorm, post-norm | cheaper, and normalizing before each sublayer trains stably at extreme depth |
Want the original attention computed by hand — real Q/K/V matrices, softmax by the numbers, and a runnable 30-line implementation? That lives on our Attention Is All You Need breakdown, worked end to end.
From a stack of blocks to text
Modern LLMs are decoder-only: a stack of these blocks with one crucial constraint — causal masking. Each token is only allowed to attend to tokens before it, never ahead. That makes the model a next-token predictor: given everything so far, what comes next?
After the final block, each position's vector is projected over the whole vocabulary and passed through a softmax to get a probability distribution over the next token. To generate, you sample a token from that distribution (temperature and top-p control how adventurous), append it to the sequence, and run the whole thing again for the next token. This loop — predict, sample, append, repeat — is autoregressive generation.
| Flavor | Attention | Used for |
|---|---|---|
| Encoder-only (BERT) | Every token sees all tokens (bidirectional) | Understanding — classification, embeddings |
| Decoder-only (GPT, Llama) | Causal — only earlier tokens | Generation — the modern LLM |
| Encoder–decoder (T5, original) | Encoder bidirectional, decoder causal + cross-attends | Translation, seq-to-seq tasks |
Attention compares every token to every other, so cost grows with the square of the sequence length — the reason long context is expensive and why serving systems cache past keys/values (the KV cache) to avoid recomputing them each step.
Why this architecture won
Two properties, together, explain the transformer's dominance. First, parallelism: because every token attends to every other in one operation (not a sequential chain), a whole sequence trains at once, saturating GPUs — which is what made training on internet-scale data feasible. Second, direct long-range connections: any token can reach any other in a single step, so relationships across long distances aren't diluted the way they were in RNNs.
And crucially, it scales predictably — make it bigger and feed it more data and it reliably gets better, with no architectural change. That combination of parallel training, expressive attention, and clean scaling is why the same block underpins GPT, Llama, Claude, and essentially every modern language model. Once you see the block, you've seen the whole family.
Worked example: sizing GPT-2 small from the block formula
The page’s napkin formula is 12·d² per block. Check it against a model whose size is public: GPT-2 small, with width d = 768, 12 blocks, 12 heads, a 50,257-token vocabulary and a 1,024-token context. Biases and norm weights are left out; they are tiny.
| Step | What | Arithmetic | Result |
|---|---|---|---|
| 1 | Attention weights, one block | 4 × 768² | 2,359,296 |
| 2 | FFN weights, one block | 8 × 768² | 4,718,592 |
| 3 | All 12 blocks | 12 × 12 × 768² | 84,934,656 |
| 4 | Token embeddings | 50,257 × 768 | 38,597,376 |
| 5 | Learned position embeddings | 1,024 × 768 | 786,432 |
| 6 | Total | steps 3 + 4 + 5 | 124,318,464 ≈ 124M |
| 7 | Attention scores, full context | 1,024² per head × 12 heads × 12 blocks | 150,994,944 per forward pass |
| 8 | KV cache per token | 2 (K and V) × 12 blocks × 768 | 18,432 numbers = 36 KiB at fp16 |
| 9 | KV cache, full context | 36 KiB × 1,024 tokens | 36 MiB per sequence |
Step 6 lands on GPT-2 small’s published size of about 124M. The output layer adds nothing because GPT-2 reuses (ties) the token-embedding matrix to score the vocabulary. Notice step 4: in a small model the embedding table is almost a third of the weights, so the “two-thirds is FFN” rule applies to the blocks, not the whole model. Steps 7 to 9 are the costs that grow with context rather than with weights. Double the context and step 7 quadruples, while steps 8 and 9 only double. That 36 MiB is per sequence, so a server batching 32 conversations at full context holds over a gigabyte of cache for this tiny model alone.
Cost and trade-offs: where the compute and memory go
Here n is the sequence length, d the model width and L the number of blocks. Costs are per block unless stated.
| Case | Time | Space | Why |
|---|---|---|---|
| Attention scores (training, prefill) | O(n²·d) | O(n²) per head if the score matrix is stored | Every token is compared with every other. This is the term that makes long context expensive. |
| Projections + FFN | O(n·d²) | O(d²) weights | Linear in n. Counting 2 FLOPs per multiply-add, weights cost 24·d² per token and attention 4·n·d, so they break even at n = 6d: 4,608 tokens for GPT-2’s d = 768. |
| Decode one token, with KV cache | O(n·d + d²) | Cache grows O(L·d) per token | The new token’s query reads every cached key and value. Speed is bounded by memory bandwidth, not arithmetic. |
| Decode without a KV cache | O(n²·d + n·d²) every step | No cache | Each new token reprocesses the whole prefix. Nobody serves this way. |
| FlashAttention | Same O(n²·d) | O(n) extra | Computes exact attention in tiles that fit on-chip, so the n×n matrix is never written to GPU memory. Faster in practice, same answer. |
| GQA (page section 6) | Same compute | KV cache ÷ (query heads ÷ KV heads) | 32 query heads on 8 KV heads cuts the cache 4×. |
| RNN, for contrast | O(n·d²) total | O(d) state | Cheaper on paper, but n steps that must run one after another, so it cannot use a GPU’s parallelism during training. |
The short version: weights dominate at short context, the n² term dominates at long context, and during generation the KV cache, not the arithmetic, sets how many users one GPU can serve.
Common mistakes
- Dropping the √d
The dot product of two d-dimensional vectors with unit-variance entries has variance d. Unscaled, scores at d = 64 are about 8× too spread out, softmax saturates to nearly one-hot, and gradients through it vanish. The
/√din the formula keeps the scores in softmax’s useful range. - Thinking more heads means more parameters
Heads split the width; they do not add to it. 12 heads of 64 dimensions use the same four d×d matrices as one head of 768. Attention stays at 4d² whatever the head count. Only the number of separate attention patterns changes.
- Forgetting the causal mask in training
Train a decoder without it and each position can see the token it is meant to predict. Loss drops to almost nothing, then generation is gibberish, because at inference the future is not there to copy. A suspiciously perfect first epoch usually means this bug.
- Letting padding into attention
Batches pad short sequences to one length. Without a padding mask, real tokens attend to pad tokens, and the same input gives different outputs depending on what it was batched with. Mask pad positions to −∞ before the softmax.
- Reading attention weights as an explanation
A head putting weight on animal is suggestive, not proof of why the model answered as it did. Values are mixed again by later layers, residual paths and the FFN. Research has shown that quite different attention patterns can produce the same prediction.
Where this shows up in production and interviews
- LLM serving
Prefill (reading the prompt) is the O(n²·d), compute-bound phase. Decode (one token at a time) is memory-bound and lives or dies by the KV cache from step 9. Paged KV caches and continuous batching exist to pack more sequences into that memory. The LLM Serving handbook works the numbers.
- Long context
The n² term is why long context is priced and engineered separately. FlashAttention removes the n×n memory cost without approximating. Sliding-window attention, used by Mistral 7B, caps how far back each layer looks, so per-token cost stops growing with n.
- Beyond text
The block does not care what a token is. Vision Transformers cut an image into 16×16-pixel patches and treat each one as a token. Whisper runs an encoder–decoder transformer over audio spectrograms. Embedding models are encoder-only stacks with one vector pooled out at the end.
- The interview version
Expect to derive
12·L·d²and check it on a known model as above, explain why attention is O(n²), size a KV cache, say why the√dis there and what the causal mask prevents. Getting the arithmetic right counts for more than naming the latest variant.
Check yourself
A model has d = 4096 and 32 heads. How wide is each head, and does moving to 64 heads change the attention parameter count?
Each head is 4096 ÷ 32 = 128 wide. At 64 heads each is 64 wide, but the Q, K, V and output matrices are still 4096×4096, so attention stays at 4d². Head count changes how the width is divided, not how many weights there are.
You double the context from 4k to 8k tokens. What happens to the attention-score work, the FFN work and the KV cache?
Score work is O(n²), so it grows 4×. FFN and projection work is O(n), so it doubles. The KV cache holds one K and V per token per block, so it doubles too. Long context gets expensive because one part of the cost grows faster than everything else.
Which adds more parameters to a 12·L·d² model: doubling the width or doubling the depth?
Doubling d multiplies the block weights by 4, because d is squared. Doubling L multiplies them by 2. That is why parameter budgets are usually spent on width and depth together rather than on one alone.
Your decoder’s training loss is near zero after one epoch, but sampling produces nonsense. What do you check first?
The causal mask. Without it each position can attend to the next token, which is the training target, so the model learns to copy instead of predict. At generation the future is not there, and the copying strategy collapses.
Quick answers
What is a transformer?
A neural network architecture built around self-attention, letting every token in a sequence directly attend to every other token. It processes sequences in parallel and captures long-range dependencies, which is why it underlies modern LLMs.
What is self-attention?
Each token produces a query, key, and value; it scores its query against every token's key, softmaxes those into weights, and takes a weighted sum of the values — gathering information from the most relevant tokens regardless of distance.
Why multiple heads?
Each head has its own Q/K/V projections and can specialize in a different relationship (syntax, coreference, position). Concatenating heads gives a richer, multi-perspective representation than a single attention.
How does it generate text?
A decoder-only transformer predicts a probability distribution over the next token, samples one, appends it, and repeats — autoregressive generation, with causal masking so each position only sees earlier tokens.
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →
- The GPU Fundamentals HandbookThe bottleneck is never where you think. SIMT and warps, the memory hierarchy from registers to HBM, arithmetic intensity and the roofline (compute-bound vs bandwidth-bound), where LLM workloads actually land, why FlashAttention was inevitable, the kernel stack from CUDA to Triton to torch.compile, and the multi-GPU interconnect story.Read →