The transformer,
and the attention that runs it.

Language is a sequence, and the meaning of a word depends on the words around it — sometimes ones far away. "The animal didn't cross the street because it was too tired" — what does it refer to? To resolve that, a model must connect it back to animal, which could be many words back.

The previous generation of models, RNNs, read a sequence one token at a time, carrying a running summary forward. That has two fatal flaws: it's sequential (you can't process token 100 until you've processed 1–99, so it can't parallelize on modern hardware), and information from far-back tokens gets diluted as it's passed hand-to-hand. The transformer's answer is radical: don't pass information along a chain — let every token directly attend to every other token, all at once.

A model can't consume raw text, so first the text is split into tokens — subword pieces (roughly, common words are one token, rare words split into several). Each token id is mapped to a learned embedding: a vector (say 768 or 4096 numbers) that places the token in a space where similar meanings sit nearby. Now the sequence is a stack of vectors.

But there's a catch. Attention, as we'll see, treats its inputs as an unordered set — it has no built-in sense of "first" or "last". "Dog bites man" and "man bites dog" would look identical. So we add positional encoding: a signal (fixed sinusoids, or a learned/rotary scheme like RoPE) added to each embedding that tells the model where each token sits. Order is injected as data, not baked into the mechanism.

For each token, the model produces three vectors by multiplying its embedding by three learned weight matrices: a query (Q — "what am I looking for?"), a key (K — "what do I offer?"), and a value (V — "what I'll pass on if chosen"). The metaphor is a lookup: your query is compared against everyone's key to decide whom to listen to.

Concretely, a token's query is dotted with every token's key to produce a relevance score for each. Those scores are scaled and passed through a softmax, turning them into weights that sum to 1 — an attention distribution. The token's new representation is the weighted sum of all the value vectors using those weights. In one line:

Attention(Q,K,V) = softmax( Q·Kᵀ / √d ) · V

Read it as: compare (Q·Kᵀ), normalize (softmax), then gather (·V). The token it can give almost all its weight to animal and pull in its meaning directly — no chain, no dilution, regardless of distance. Every token does this simultaneously, which is why the whole thing runs in parallel.

One attention operation forces the model to blend all kinds of relationships into a single set of weights. That's limiting — a word relates to others in many ways at once (grammatical subject, what it modifies, what pronoun refers to it). So transformers run several attention operations in parallel, called heads, each with its own learned Q/K/V projections.

Because each head has different projection matrices, each learns to focus on a different kind of relationship. One head might track the previous word, another might link verbs to their subjects, another might resolve pronouns. The heads' outputs are concatenated and mixed by another matrix, giving the model a rich, multi-perspective view where a single attention would give one flat summary.

Attention lets tokens exchange information; the model still needs to process what each token gathered. So each transformer block pairs attention with a small per-token feed-forward network (FFN) — two linear layers with a non-linearity — that transforms each token's vector independently. Attention mixes across tokens; the FFN thinks about each one.

Two more pieces make deep stacks trainable. Residual connections add each sublayer's input to its output, so information (and gradients) can skip layers — you're always refining the representation, never rebuilding it. Layer normalization keeps the numbers in a stable range so training doesn't blow up. The block is just: attention → add & norm → FFN → add & norm.

That's the entire architecture. "Scaling a transformer" mostly means: bigger embeddings, more heads, more of these blocks, trained on more data. The block never changes.

Modern LLMs are decoder-only: a stack of these blocks with one crucial constraint — causal masking. Each token is only allowed to attend to tokens before it, never ahead. That makes the model a next-token predictor: given everything so far, what comes next?

After the final block, each position's vector is projected over the whole vocabulary and passed through a softmax to get a probability distribution over the next token. To generate, you sample a token from that distribution (temperature and top-p control how adventurous), append it to the sequence, and run the whole thing again for the next token. This loop — predict, sample, append, repeat — is autoregressive generation.

Two properties, together, explain the transformer's dominance. First, parallelism: because every token attends to every other in one operation (not a sequential chain), a whole sequence trains at once, saturating GPUs — which is what made training on internet-scale data feasible. Second, direct long-range connections: any token can reach any other in a single step, so relationships across long distances aren't diluted the way they were in RNNs.

And crucially, it scales predictably — make it bigger and feed it more data and it reliably gets better, with no architectural change. That combination of parallel training, expressive attention, and clean scaling is why the same block underpins GPT, Llama, Claude, and essentially every modern language model. Once you see the block, you've seen the whole family.