Build an LLM From Scratch

What each step gives you

1. The Tokenizer

   Type anything and watch it split into sub-word tokens with integer IDs, race character, word and sub-word tokenizers, then build byte-pair encoding by merging the most frequent pair on a tiny corpus. It answers the first question of the path, what the model actually reads, and explains practical effects such as token cost, context limits and why models miscount the letters in strawberry.

2. BPE Merge Step

   Implement one BPE training step: count every adjacent symbol pair across the corpus, pick the most frequent with a lexicographic tie-break, and merge every non-overlapping occurrence from left to right. Repeat that step a few thousand times and you have trained a tokenizer, which prepares you to discuss vocabulary-size trade-offs and why the same word can split differently in different models.

3. The Embeddings Handbook

   Tokens become vectors arranged so that similar meaning lands nearby, and this handbook covers why the angle between vectors is the similarity, why you normalize and then take a dot product, what extra dimensions buy, and the pitfalls of embeddings in real search systems. It gives you the vocabulary for the embedding lookup at the bottom of every Transformer and for the cosine questions that follow.

4. Word2Vec

   The 2013 paper that learned dense word vectors from the company words keep, through skip-gram and CBOW, and made king minus man plus woman landing near queen famous. The breakdown is candid that the analogy arithmetic works less cleanly than the famous examples suggest, and its core limit, one vector per word whatever the context, is the gap later contextual models were built to close.

5. The Meaning Map: Embeddings

   Click through a map of twelve words where royalty, animals and places cluster, rank each word's nearest neighbors by cosine, and run analogies such as king minus man plus woman or Paris minus France plus Japan. It turns the previous two steps into geometry you can see, which helps when you need to explain why cosine ignores vector length and how concepts become directions.

6. Seq2Seq

   The 2014 encoder-decoder paper that replaced a translation pipeline with one network, and its weakness: the whole input squeezed into a single fixed-length vector. Knowing that bottleneck makes attention feel necessary rather than arbitrary, and the paper's reversed-input trick and its case for beam search over greedy decoding still come up in questions about generation.

7. Attention Is All You Need

   With the bottleneck in mind, the 2017 paper replaces recurrence with self-attention: queries, keys and values, the scaled dot-product formula computed by hand, multi-head attention, positional encoding and the full encoder-decoder architecture. Read it as the blueprint for every later step in this path, and run its 30-line Python self-attention so the next challenge is a translation exercise rather than a puzzle.

8. Single-Head Attention

   Write scaled dot-product attention yourself: score each query row against every key, divide by the square root of d, softmax each row into weights, and return the weighted sum of the value rows. Passing the hidden tests prepares you for whiteboard requests to implement attention in NumPy and for the question of why the scaling factor is there at all.

9. The Spotlight: Attention

   Click any word in a short sentence and see how its attention splits across the others, with a verb leaning on its subject and a determiner pointing at its noun, as weights that sum to one. Use it right after writing attention to connect your score matrix to what heads are said to learn, remembering that the page's numbers are a hand-picked illustration rather than output from a trained model.

10. Softmax

    Softmax runs inside every attention head and once more at the output, where it turns logits into a next-token distribution, so here you write the stable version that subtracts the maximum before exponentiating. Once it passes, connect it back to the model: dividing logits by a temperature before this function is exactly how sampling becomes sharper or flatter.

11. Layer Normalization

    Implement layer normalization for one vector: subtract the mean, divide by the square root of the variance plus an epsilon, then apply a learned per-element scale and shift. It is the stabilizer wrapped around every Transformer sub-layer, and writing it prepares you for questions about why deep stacks need normalization, what the epsilon protects against, and how layer norm differs from batch norm.

12. Inside a Transformer: Self-Attention

    Pick a query token in the four-token sentence The robot picked it and step through the arithmetic: scores from query-key dot products scaled by the square root of d, softmax weights, and the context vector as a weighted sum of values. Watching it attend most to robot with real numbers is a check that your challenge code and your mental model agree.

13. The Transformers Handbook

    The consolidation step: tokens, embeddings and position, self-attention and multi-head attention, then the transformer block itself, with its feed-forward network, residual connections and layer normalization, a block's parameter budget and what has changed since 2017, and how a stack of blocks produces text. After the pieces you have built, this is where they assemble into one forward pass you should be able to draw and narrate end to end.

14. RoPE (RoFormer)

    Attention is order-blind, and RoPE's answer is to rotate queries and keys by an angle that depends on position, so their dot product depends only on the distance between tokens. The breakdown compares absolute and relative schemes, shows relative position falling out of the rotation, and covers frequencies and long context, the background for questions about how LLaMA-style models handle position.

15. GPT-3

    Scaling a decoder-only Transformer to 175 billion parameters made few-shot, in-context learning appear without any fine-tuning, and this breakdown derives the 175B figure by hand and lets you size and price any GPT on a napkin. It prepares you to explain zero-, one- and few-shot prompting, how GPT-3 differs from BERT, and the limitations its authors admitted.

16. Scaling Laws

    Kaplan and colleagues showed loss falling as a smooth power law in parameters, data and compute, which turned model sizing into a forecast. The page works the laws with real numbers, explains how to spend a fixed compute budget, and covers how Chinchilla later revised the recommended balance toward more data, so you can explain both why pretraining keeps paying off and where the original advice changed.

17. Cross-Entropy Loss

    Implement the loss a language model trains against: the negative log-probability assigned to the true class, for either an index or a one-hot target, with an epsilon guard so a zero probability gives a large finite loss instead of infinity. It connects softmax to training and prepares you to explain perplexity, label smoothing, and why the loss punishes confident mistakes so heavily.

18. The Fine-Tuning Handbook

    With a pretrained model in hand, this handbook covers how to adapt it: when to fine-tune instead of prompting or retrieval, full fine-tuning versus parameter-efficient methods, how a LoRA update works and how few parameters it really trains, data quality, catastrophic forgetting and evaluation. It prepares you to size a LoRA run yourself and to defend that choice in a design discussion.

19. InstructGPT

    InstructGPT's three steps, supervised fine-tuning, a reward model trained on human rankings, and reinforcement learning with PPO, are how a raw next-token predictor became an assistant that follows instructions. The breakdown includes training a reward model from preferences and the result that a much smaller aligned model was preferred over GPT-3, which sits at the center of any question about what RLHF changes.

20. The Taste Trainer: RLHF & DPO

    You play the labeler, picking the better of two answers until a reward model predicts your taste, then turn up the optimization pressure and watch the policy hack the reward while a KL penalty pulls it back. The final act compares the classic pipeline with DPO, which trains directly on preference pairs, so you can explain why DPO became a common default for preference tuning.