LEARNING PATH · AI Engineering

The Papers Bootcamp

For engineers who want to read the canonical deep-learning papers in the right order.

Advanced ~5h 30m22 lessons22 steps

The papers that built modern AI, sequenced so each one earns the next — from word2vec and the Transformer through scaling laws, RLHF, chain-of-thought, RAG, LoRA and MoE, out to vision and diffusion. Every entry is a plain-English breakdown, not the raw PDF.

Read the canonical papers in a coherent, building order
Connect each paper to the capability it unlocked
Trace the line from word vectors to reasoning models
Follow the field beyond language into vision and diffusion

0 / 22 done · 0%

How this path works

This path is for engineers who keep meeting paper names in model cards, docs and interviews (RLHF, LoRA, mixture of experts, chain-of-thought) and want to understand where each idea came from, in order, instead of from scattered summaries. It is marked Advanced because it moves through twenty-two papers quickly, but every entry is a plain-English breakdown with the key numbers worked out, so you do not need the original PDFs to follow it.

Reading a paper well means answering four questions: what problem it attacked, what it changed compared with the work before it, what evidence it offered, and how the claim held up afterward, including what the authors themselves admitted. Each breakdown is organized around those questions, and this path puts the papers in an order where each answer sets up the next question.

It starts with representations and sequence models, because the Transformer is easiest to understand as the fix for the seq2seq bottleneck. The encoder and decoder lines follow, then the two scaling papers, with Chinchilla placed after the Kaplan result it revised. Next come the papers that made models useful (RLHF, reasoning by prompting, retrieval, cheap fine-tuning and simpler alignment), then efficiency, agents and trained reasoning, and finally vision, image generation and speech.

What each step gives you

Word2Vec
The 2013 paper that turned words from arbitrary IDs into dense vectors learned from the company they keep, using the skip-gram and CBOW objectives, and made analogy arithmetic on meaning work. The breakdown lets you do that arithmetic yourself and connects it to modern embeddings and RAG, so you begin with the idea every later paper assumes: meaning as geometry.
Seq2Seq
Sutskever, Vinyals and Le showed in 2014 that a single encoder-decoder network could map a variable-length input sequence to a variable-length output, replacing a pipeline of separately built parts. The breakdown covers the tricks that made it train, including reversing the input and using beam search over greedy decoding, plus the fixed-vector bottleneck the next paper exists to remove.
Attention Is All You Need
The 2017 paper that dropped recurrence entirely: self-attention lets every position look at every other in parallel, multi-head attention runs several of those comparisons at once, and positional encodings restore word order. Read straight after seq2seq, it answers that bottleneck directly, and the breakdown's hand-computed attention and short Python version make the rest of the bootcamp far easier to read.
BERT
BERT made pre-training the default for language tasks by reading text in both directions with masked language modeling and next-sentence prediction, then fine-tuning the same model for each task. The breakdown derives its 110M and 340M parameter counts by hand and explains the masking recipe, which prepares you to say clearly how an encoder like BERT differs from a decoder like GPT.
GPT-3
GPT-3 scaled a decoder-only Transformer to 175 billion parameters and showed that, at that size, a model can pick up a task from a few examples in the prompt without any fine-tuning. The breakdown separates zero-shot, one-shot and few-shot prompting, derives the parameter count, and is candid about what the model could not do, which is the honest frame for in-context learning.
Scaling Laws
Kaplan and colleagues found that language-model loss falls as a smooth power law in parameters, data and compute, which turned model sizing from a bet into a forecast. The breakdown works the laws with real numbers and walks through the paper's advice on spending a fixed compute budget, so you can predict a model's loss before training it and see exactly which conclusion the next paper revisits.
Chinchilla
Chinchilla measured the same trade-off and reached a different answer: for a fixed compute budget, parameters and training tokens should grow together, roughly twenty tokens per parameter, which meant earlier large models were undertrained. The breakdown re-derives the result from Gopher's budget and explains how a 70B model beat a 280B one, the clearest case in the bootcamp of a finding being revised.
InstructGPT
InstructGPT changed the objective from predicting text to following instructions, in three stages: supervised fine-tuning on demonstrations, a reward model trained on human preference rankings, and reinforcement learning with PPO against that reward. The breakdown explains why labelers preferred a much smaller InstructGPT over GPT-3 and what RLHF does not solve, background you need for DPO later in the path.
Chain-of-Thought Prompting
Showing a model a few worked examples that include their reasoning steps let it solve multi-step arithmetic and logic problems it failed when asked for the answer directly, and the effect appeared mainly in large models. The breakdown also covers the zero-shot variant that simply asks the model to reason step by step, and connects the idea to today's reasoning models.
Self-Consistency
Instead of trusting one chain of thought, sample many reasoning paths, extract each final answer, and take a majority vote, a decoding change that raised accuracy substantially on reasoning benchmarks. The breakdown simulates the vote climbing toward the right answer, explains what the extra samples cost, and links the method to the wider idea of spending more compute at inference time.
Retrieval-Augmented Generation
The 2020 retrieval-augmented generation paper paired a dense passage retriever with a sequence-to-sequence generator, so a model could look facts up instead of answering only from its weights. The breakdown explains both halves, the difference between RAG-Sequence and RAG-Token, and why retrieval reduces hallucination, which makes it the research root of the RAG systems engineers build today.
LoRA
LoRA freezes a pretrained model and trains two small low-rank matrices for each adapted weight, based on the observation that the change a new task needs has low rank. The breakdown works the parameter savings with real numbers, shows why merged adapters add no inference latency, and explains rank and QLoRA, the questions to settle before you decide to fine-tune.
Direct Preference Optimization
Direct Preference Optimization removes the separate reward model and the reinforcement-learning loop from RLHF, training the language model directly on preference pairs with a single loss. Read it against InstructGPT's three stages: the breakdown works the loss with real numbers, explains why training is more stable than with PPO, and spells out what preference data the method needs.
Mixture of Experts
Sparsely gated mixture-of-experts layers hold many expert sub-networks and let a gating network route each token to only a few of them, so total parameters grow without a matching growth in compute per token. The breakdown counts the parameters that actually fire, explains load balancing as the catch, and prepares you to read an MoE model's total and active parameter counts correctly.
FlashAttention
FlashAttention made exact attention faster and far lighter on memory by treating it as a data-movement problem between slow GPU memory and fast on-chip SRAM, solved with tiling and an online softmax. The breakdown works the tiling math, stresses that the result is exact rather than approximate, and explains how the method helped make long context windows practical.
ReAct
ReAct interleaves reasoning traces with actions: the model thinks, calls a tool such as search, reads the observation and thinks again, so its reasoning gets checked against the world instead of compounding its own errors. The breakdown walks a real thought, action and observation trace and lets you run the loop yourself, the same pattern agent loops still follow.
Toolformer
Toolformer taught a language model to insert API calls into text on its own, without human labels: a candidate call was kept only if its result made the following text easier to predict. The breakdown explains that filter, the three-step recipe and the tools it learned, and contrasts trained tool use with ReAct's prompted version, which clarifies what function calling is doing.
DeepSeek-R1
DeepSeek-R1 showed that long step-by-step reasoning can emerge from reinforcement learning on outcome rewards alone, in the R1-Zero model, then smoothed its rough edges with cold-start data and distilled the reasoning into smaller models. The breakdown covers the self-correcting behavior that appeared during training, what was published openly, and the limits, closing the line from prompted to trained reasoning.
Vision Transformer (ViT)
The Vision Transformer cut an image into 16 by 16 pixel patches, embedded each one as a token and fed them to a standard Transformer encoder, trading the built-in assumptions of convolutional networks for more training data. The breakdown explains the class token, why ViT needs large datasets to beat CNNs, and how to price attention over patches, the field's first step beyond text on this path.
CLIP
CLIP trained an image encoder and a text encoder together on hundreds of millions of image-caption pairs from the internet, so matching pictures and descriptions land close together in one shared space. The breakdown builds the contrastive matrix, shows zero-shot classification by comparing an image against written labels, and explains how that shared space feeds text-to-image models.
DDPM
DDPM made diffusion models work by fixing a forward process that adds noise on a schedule and training a network to predict that noise, so generation runs the process in reverse starting from pure noise. The breakdown works the noise schedule with real numbers, diffuses and recovers a signal in about 25 lines of code, and compares diffusion with GANs.
Whisper
Whisper traded clean labels for scale, training a plain encoder-decoder Transformer on 680,000 hours of weakly supervised audio, and proved far more robust on unfamiliar recordings than models tuned to one benchmark. The breakdown covers the spectrogram input, how special tokens steer one model across transcription, translation and language identification, and its limits, ending the bootcamp in speech.

A study plan

Week 1
Read Word2Vec, Seq2Seq and Attention Is All You Need, in that order. After each one, write three sentences before moving on: the problem, what changed against the paper before it, and what the authors admitted. Answer each page's question set without looking, then check.
Week 2
Read BERT, GPT-3, Scaling Laws and Chinchilla. Explain the encoder and decoder lines to someone in two minutes, then write one paragraph on what Chinchilla changed about the Kaplan recommendation for spending a compute budget.
Week 3
Read InstructGPT, Chain-of-Thought, Self-Consistency and RAG, four different ways to make a pretrained model more useful. For each, note what it costs (human labels, extra tokens, extra samples or a retriever) and what it fixes, since that trade-off is how you will compare them in practice.
Week 4
Read LoRA, DPO, Mixture of Experts and FlashAttention. Redo one worked calculation from each breakdown with your own numbers: a LoRA parameter count, a DPO loss, the active parameters of an MoE layer, or the tiling arithmetic behind FlashAttention.
Week 5
Read ReAct, Toolformer and DeepSeek-R1, which together trace the move from prompted reasoning to trained tool use to reasoning learned through reinforcement learning. Run the reason-and-act loop on the ReAct page, then describe in your own words how Toolformer decides which tool calls to keep.
Week 6
Finish with Vision Transformer, CLIP, DDPM and Whisper. For each, name the idea it borrowed from language models and what had to change for images or audio, then reread your Week 1 notes to see how far the same few ideas traveled.

Questions about this path

Which AI papers should I read first?

Start with the papers the rest depend on: Word2Vec for the idea of embeddings, Seq2Seq for the encoder-decoder and its fixed-vector bottleneck, and Attention Is All You Need for the Transformer that removed that bottleneck. After those three, BERT and GPT-3 show the two directions the Transformer took, and the scaling papers explain why size kept paying off. Reading in this dependency order matters more than reading everything, because each paper's contribution is easiest to see against the one it improved on.

How do you read a machine learning paper effectively?

Read with four questions in mind rather than front to back. What problem does it attack, and why did earlier methods fail at it? What exactly changed? What evidence supports the claim, and on which benchmarks? What do the authors admit, and what did later work confirm or overturn? Skim the abstract and conclusion first, then work through one central equation or example by hand. Writing a three-sentence summary afterward is the quickest test of whether you understood it.

Do I need to read the original PDFs, or are breakdowns enough?

Breakdowns are enough to understand what each paper contributed and to discuss it confidently, which is what most engineering work and interviews need. Go to the original when you need exact experimental settings, precise numbers, ablations or the trail of related work, or when the paper bears directly on something you are building. A good habit is to read the breakdown first so you know the main idea, then open the PDF and read only the sections that answer your remaining questions.

How much math do I need to understand deep learning papers?

Less than the notation suggests. Most of the papers in this path rest on a handful of tools: vectors and dot products, matrix multiplication, softmax, logarithms in a loss function, and gradients for training. Probability matters more for RLHF, DPO and diffusion, and a little statistics helps you judge the reported results. The breakdowns work their key numbers by hand, so you can follow the argument first and pick up the formal math where a specific paper requires it.

The canonical deep-learning papers make far more sense read in the right order, each building on the last. This path sequences them — from the foundations through attention, scaling, and modern techniques — so you build a connected mental model of how the field actually got here, instead of a pile of disconnected PDFs.

← All learning paths

Word2Vec

Seq2Seq

Attention Is All You Need

BERT

GPT-3

Scaling Laws

Chinchilla

InstructGPT

Chain-of-Thought Prompting

Self-Consistency

Retrieval-Augmented Generation

LoRA

Direct Preference Optimization

Mixture of Experts

FlashAttention

ReAct

Toolformer

DeepSeek-R1

Vision Transformer (ViT)

CLIP

DDPM

Whisper