Build an LLM From Scratch
For engineers who want to understand a language model down to the matrix multiply.
No black boxes. Start at raw bytes and end at an aligned assistant — tokenization, embeddings, attention, the full forward pass, the training loss, and RLHF — each idea paired with a runnable challenge or lab that makes you build the piece by hand.
- Explain every stage from tokens to a trained, aligned model
- Implement attention, softmax, layer-norm and cross-entropy by hand
- Reason about scaling laws and why pretraining works
- Understand how RLHF turns a raw LM into an assistant
The Tokenizer
Text becomes tokens: watch BPE build a vocabulary from raw bytes.
BPE Merge Step
Implement the merge step every BPE tokenizer is built on.
The Embeddings Handbook
Tokens become vectors — what an embedding actually is.
Word2Vec
The paper that made “king − man + woman ≈ queen” real.
The Meaning Map: Embeddings
Fly through an embedding space and see meaning as geometry.
Seq2Seq
Before attention: encoder→decoder and its bottleneck.
Attention Is All You Need
The Transformer — the architecture every LLM inherits.
Single-Head Attention
Implement scaled dot-product attention from scratch.
The Spotlight: Attention
Watch attention weights light up token to token.
Softmax
The function at every attention and output head.
Layer Normalization
The normalization that keeps deep stacks trainable.
Inside a Transformer: Self-Attention
Push a sequence through a full forward pass, layer by layer.
The Transformers Handbook
Consolidate: the whole architecture in one handbook.
RoPE (RoFormer)
How modern models encode position — rotary embeddings.
GPT-3
Scale it up: few-shot learning emerges.
Scaling Laws
Why more parameters and data predictably help.
Cross-Entropy Loss
The loss you train a language model against.
The Fine-Tuning Handbook
Adapt a pretrained model to your task.
InstructGPT
Turn a raw LM into an assistant — RLHF.
The Taste Trainer: RLHF & DPO
Practice the preference-tuning loop by hand.