The papers that
built modern AI.
Narrated, animated breakdowns of the foundational research papers behind today's AI — self-attention, embeddings, and the ideas everything else builds on. Free to watch.
Attention Is All You Need
The paper that introduced the Transformer — the architecture underneath GPT, BERT, Claude, and Gemini. Self-attention, Query/Key/Value, multi-head attention, positional encoding, and the real BLEU results, explained in depth with analogies and animated visuals, not just equations.
BERT
The paper that made "pre-train once, fine-tune anywhere" the default for NLP. Bidirectional context, masked language modeling, next-sentence prediction, the [CLS]/[SEP] tokens, and the GLUE sweep — explained with analogies, not just equations.
Retrieval-Augmented Generation
The paper behind every "chat with your docs" product. Pair a generator with a neural retriever so the model looks knowledge up instead of memorizing it — dense retrieval, top-k documents, RAG-Sequence vs RAG-Token, and why retrieval cuts hallucination.
Chain-of-Thought Prompting
Show a model a few worked examples with their reasoning and it starts solving problems it used to fail. Why reasoning emerges only at scale, the GSM8K jump, zero-shot "Let's think step by step", and how CoT seeded modern reasoning models.
LoRA
The paper that made fine-tuning affordable. Freeze the giant pretrained model and train two tiny low-rank matrices instead — up to 10,000x fewer trainable parameters, no extra inference latency, and swappable per-task adapters. The road to QLoRA and PEFT.
ReAct
The blueprint for AI agents. Interleave reasoning with actions so a model can think, use a tool, observe the result, and think again — the Thought–Action–Observation loop that grounds reasoning, cuts hallucination, and underpins every tool-using agent.
Direct Preference Optimization
RLHF without the RL. Align a model to human preference pairs with one simple classification-style loss — no separate reward model, no unstable reinforcement learning. Why "your LM is secretly a reward model", and how DPO became the default alignment method.
Chinchilla
The compute-optimal scaling law that said models were too big. Parameters and training tokens should scale together — ~20 tokens per parameter — so a 70B model trained on 1.4T tokens beat the 280B giants. Why every model since trains on trillions of tokens.
FlashAttention
Attention was slow for the wrong reason. By treating it as a memory-movement problem — tiling, online softmax, keeping work in fast SRAM instead of slow HBM — FlashAttention makes exact attention several times faster and cuts its memory from quadratic to linear, unlocking long context windows.
Mixture of Experts
How models got huge without getting slow. Many expert sub-networks plus a gating network that routes each token to just a few — so a model holds hundreds of billions of parameters but only activates a fraction per token. Gating, top-k routing, load balancing, and the road to Switch Transformer and Mixtral.
Word2Vec
How words became math. Dense word vectors learned from the company a word keeps, so king − man + woman lands near queen. Skip-gram, CBOW, the distributional hypothesis, and the origin of every embedding model powering search and RAG.
ResNet
The skip connection that made deep learning deep. Why stacking layers made networks worse, and how residual learning let them reach 150+ layers and win ImageNet — the identity shortcut every Transformer still uses inside every block.
Diffusion Models
How AI paints from pure noise. Add noise to an image step by step until it's static, then train a network to reverse it and sculpt pictures out of chaos — the forward and reverse processes behind Stable Diffusion, DALL·E, and Midjourney.
CLIP
How AI learned to connect images and words. Train an image encoder and a text encoder together on 400M internet image-caption pairs so pictures and words share one space — enabling zero-shot classification and the text-to-image bridge behind modern generators.
GPT-3
The paper that made models learn from the prompt. Scale a Transformer decoder to 175 billion parameters and a new behavior emerges — in-context learning: teach it a task with a few examples in the prompt, no fine-tuning, no weight updates. Zero-, one-, and few-shot prompting explained.
InstructGPT
How RLHF taught models to follow instructions — the paper behind ChatGPT. The three-step recipe: supervised fine-tuning, a reward model learned from human rankings, and PPO reinforcement learning. A 1.3B aligned model beat 175B GPT-3 on human preference — alignment over scale.
Llama 2
The paper that opened up the LLM. Meta released a strong family of open-weight models — 7B, 13B, 70B — with chat alignment included. Pretraining on 2T tokens, grouped-query attention, and Llama 2-Chat's RLHF with separate helpfulness and safety reward models — the release that seeded the open-model ecosystem.
Scaling Laws
The physics of making models bigger. A language model's loss falls as a smooth power law in model size, dataset size, and compute — so performance is predictable before you train. Compute-optimal training, the forecast that justified GPT-3, and how Chinchilla later corrected the recipe.
T5
The model that turned every NLP task into text-to-text. One encoder-decoder Transformer, one loss, one format: translation, summarization, classification, and Q&A all become text-in, text-out. The C4 dataset, span-corruption pre-training, and a rigorous study of what drives transfer learning.
Vision Transformer (ViT)
When an image became 16×16 words. Cut a picture into patches, treat each patch as a token, and feed them to a plain Transformer — no convolutions. Patch embeddings, the class token, and why data scale lets attention beat the CNNs that ruled computer vision for a decade.
Latent Diffusion
The paper behind Stable Diffusion. Compress an image into a small latent space with a VAE, run the diffusion denoising there instead of on pixels for a tens-of-times efficiency win, and steer it with text via cross-attention. The architecture that put text-to-image generation on consumer GPUs.
DDPM
The original denoising diffusion math — the paper that made diffusion work. A fixed forward process adds Gaussian noise over many steps; a U-Net learns the reverse. The masterstroke: train the network to simply predict the added noise with a plain MSE loss. The foundation under every modern image generator.
GANs
The counterfeiter and the detective. Two networks locked in a game: a generator forging fakes from noise and a discriminator trying to catch them. The minimax objective that taught machines to imagine, its notorious instabilities and mode collapse, and how diffusion later overtook it.
AlexNet
The paper that started the deep learning revolution. AlexNet won ImageNet 2012 by a landslide — cutting top-5 error from ~26% to ~15% — proving deep CNNs with big data and GPUs beat decades of hand-engineered features. ReLU, dropout, GPU training, and data augmentation, explained.
Seq2Seq
Teaching neural nets to translate. An encoder LSTM reads an input sentence into a single fixed vector; a decoder LSTM writes the translation out. The architecture that made neural machine translation work — and whose fixed-vector bottleneck directly motivated the attention mechanism.
Mamba
The linear-time challenger to attention. A selective state-space model that compresses history into a fixed-size state and scales linearly with sequence length — matching Transformers on language while being far cheaper on very long sequences. Selective SSMs, the recurrent/parallel duality, and hardware-aware design.
Whisper
Robust speech recognition from the open web. An encoder-decoder Transformer trained on 680,000 hours of diverse, weakly-labeled web audio — transcribing and translating dozens of languages robustly, often zero-shot. The data bet, the multitask token format, and why scale beat clean supervision.
Segment Anything (SAM)
A foundation model for cutting out objects. Point, click, or box anything in any image and SAM returns a precise mask — zero-shot. The image-encoder / prompt-encoder / mask-decoder design, and the data engine that bootstrapped SA-1B, a dataset of 1 billion masks.
DeepSeek-R1
How a model learned to reason from reward alone. Reward only correct final answers and step-by-step chain-of-thought emerges on its own — including a spontaneous "aha moment" of self-correction. R1-Zero's pure RL, cold-start data, and cheap distillation of reasoning into small open models.
Toolformer
How a language model taught itself to use APIs. Toolformer learns, self-supervised, to call a calculator, search, QA, translation, and calendar — inserting API calls into text and keeping only the calls that make its next-word prediction better. The recipe that seeded modern tool-using AI agents.
GRPO (DeepSeekMath)
The RL algorithm behind DeepSeek-R1. PPO needs a critic as large as the model itself; GRPO deletes it — sample a group of answers, score them, and reward whoever beats the group average. Half the memory, none of the critic's instability, and the engine that later made reasoning emerge from reward alone.
Titans
Models that memorize at test time. A neural long-term memory keeps learning during inference — writing what surprises it into its own weights, with momentum and adaptive forgetting — so attention handles the recent window while the memory holds the far past. Beats Transformers and Mamba on 2M+ token recall.
Constitutional AI
Alignment from principles, not labels. Write the values down as a constitution, have the model critique and revise its own outputs against them, then run RL on AI-judged preferences (RLAIF) — harmlessness that scales with compute instead of crowds, and refuses without stonewalling. The backbone of Claude's alignment.
Self-Consistency
Sample many diverse reasoning paths, marginalize out the reasoning, majority-vote the answer. Correct answers are attractors — many routes converge on them while scattered errors disagree. +17.9 points on GSM8K from a pure decoding change, and the first clean proof that inference compute is a quality dial.
Tree of Thoughts
When LLMs learn to search. Frame problem-solving as a tree of intermediate "thoughts": propose candidates, self-evaluate their promise, expand with BFS/DFS, backtrack from dead ends. Game of 24 went 4% → 74% with the same GPT-4 — the inference procedure, not the model, was the ceiling.
QLoRA
Fine-tune a 65B model on one GPU. Freeze the base in 4-bit NF4 (levels at the quantiles of a normal distribution), quantize the quantization constants, page optimizer spikes to CPU, and train 16-bit LoRA adapters on top — matching full 16-bit fine-tuning quality. Guanaco hit 99.3% of ChatGPT after 24h on a single card.
MemGPT
An operating system for LLM memory. Context window as RAM, external storage as disk, and the model as its own memory manager — paging facts in and out with function calls, revising core memories when facts change, and summarizing under memory pressure. The architecture behind modern agent memory (and Letta).
Reflexion
Agents that learn from their own mistakes — reinforcement through words instead of weights. Fail, write a verbal reflection on why, store it in episodic memory, retry with the lesson in context. Actor + Evaluator + Reflector took GPT-4 from 80% to 91% on HumanEval, and the pattern now lives inside every self-correcting agent.
Voyager
The Minecraft agent that never stops learning. An automatic curriculum picks the next just-hard-enough goal, the agent writes code to achieve it, debugs against environment feedback, and saves every verified program to a compounding skill library. 3.3× more items, 15.3× faster tech tree — and the blueprint for agents that build their own tools.
Generative Agents
The Smallville paper: 25 LLM characters living in a simulated town. A memory stream retrieved by recency × importance × relevance, reflection trees that turn events into beliefs, and plans that bend to interruptions — producing a Valentine's party that organized itself. The memory architecture behind modern persistent agents.
DeepSeek-V3
Frontier performance at a tenth of the cost. A 671B MoE with 37B active, multi-head latent attention shrinking the KV cache ~10×, auxiliary-loss-free expert balancing, multi-token prediction, and FP8 training with zero loss spikes — 14.8T tokens for ≈$5.6M of compute. The paper that reset the industry’s cost assumptions.
Kimi K2
A trillion parameters built for agents. A 1T-param open MoE (32B active) trained through 15.5T tokens with ZERO loss spikes via MuonClip (Muon + QK-clip), plus large-scale synthesis of tool-use episodes so agentic behavior is pretrained, not bolted on. The moment open weights led on agent workloads.
Adam
The optimizer that trains almost every modern network. Momentum as the first moment, RMSProp as the second, and the bias correction most explanations skip — plus the famously robust defaults (0.9 / 0.999 / 1e-8) and AdamW. Built up from scratch with worked math and runnable code you can edit in the browser.
Dropout
Randomly switch off half your neurons on every training step — and overfitting collapses. Why co-adaptation hurts, how inverted dropout scales survivors to keep expectations unchanged, and why the whole thing is secretly an ensemble of 2ⁿ networks. Worked math plus runnable code you can edit in the browser.
Batch Normalization
The layer that made deep networks train fast and forgivingly. Normalize each feature across the batch to zero mean and unit variance, then a learnable scale and shift give the network its freedom back. Internal covariate shift, the train-vs-test running-statistics gotcha, and why it unlocks higher learning rates — worked math plus runnable code.
LayerNorm & RMSNorm
The normalization inside every transformer. LayerNorm normalizes across one example’s features (batch-independent, unlike batch norm); RMSNorm drops the mean-centering to go faster. The exact invariances that separate them, and the tidy case where they’re mathematically identical — worked math plus runnable code.
Speculative Decoding
Make a big model generate 2–3× faster with zero change to its output. A tiny draft model guesses several tokens ahead; the target verifies them all in one parallel pass and keeps the correct prefix. Why decoding is memory-bound, why the output is provably identical, and the expected-tokens math with its optimal draft length — worked math plus runnable code.
RoPE (RoFormer)
The position scheme inside LLaMA, GPT-NeoX, and most modern LLMs. Instead of adding a position vector, RoPE rotates query and key by an angle set by their position — and because rotations add, the attention score depends only on relative distance. The geometry, the per-dimension frequencies, and long-context scaling — worked math plus runnable code.
ColBERT
Keep one vector per token, not one per document, and score by late interaction: each query token takes its best match anywhere in the document (MaxSim), summed. It recovers the term-level precision single-vector retrieval blurs away, at index-friendly speed. The math, the case where it clearly wins, and how it sits between dense and cross-encoder retrieval — with runnable code.
Sentence-BERT
The model that turned BERT into fast, comparable sentence embeddings — the ancestor of every embedding model behind vector search and RAG. Why plain BERT can’t be compared without a pass per pair, the siamese fix with mean pooling, and the O(n²)→O(n) arithmetic that took a task from 65 hours to 5 seconds. Worked math plus runnable code.
Switch Transformer
The mixture-of-experts model that scaled to 1.6 trillion parameters by routing each token to a single expert. Why top-1 routing simplifies MoE, the load-balancing auxiliary loss that keeps experts evenly used, expert capacity and token dropping, and how sparse activation decouples parameters from compute per token — worked math plus runnable code.
PagedAttention (vLLM)
The idea that made LLM serving several times cheaper: store the KV cache in fixed-size blocks like an OS pages memory, instead of reserving each request’s maximum length. Why contiguous reservation wastes ~95% of the cache, how paging caps the waste to one block, copy-on-write prefix sharing, and the throughput vLLM unlocks — worked math plus runnable code.
Masked Autoencoders
BERT for images, finally working. Mask 75% of an image’s patches, encode only the visible quarter with a ViT, and reconstruct the missing pixels with a light decoder. Why images need such a high mask ratio, the asymmetric design that makes the encoder ~16× cheaper, and the strong transfer results — worked math plus runnable code.
SimCLR
Contrastive learning that rivaled supervised vision with no labels. Two augmented views of an image are a positive pair against a batch of negatives; the NT-Xent loss pulls positives together and pushes negatives apart. The temperature, why augmentation and big batches are decisive, and how contrastive compares to masked pre-training — worked math plus runnable code.
Longformer
The attention pattern that let transformers read long documents. Full self-attention is quadratic, so it caps out around 512 tokens; Longformer’s sliding window makes it linear in length, with dilation and depth for reach and a few global tokens for the big picture. The sparsity math and receptive-field growth — worked math plus runnable code.
Medusa
Speculative decoding without a separate draft model. Medusa bolts extra prediction heads onto the model, each guessing a token further ahead, and verifies many candidate continuations in one pass via tree attention. Why head accuracy fades with distance, the tokens-per-pass math, and the trade-offs vs draft-model speculation — worked math plus runnable code.
Let's Verify Step by Step
Reward every reasoning step, not just the final answer. Process reward models beat outcome reward models at judging hard math — they catch right-answer/wrong-reasoning, localize the failing step, and rerank many candidate solutions to pick sound ones. The process-vs-outcome math and the PRM800K dataset — worked math plus runnable code.
Product Quantization
The compression that makes billion-scale vector search fit in memory. Split a vector into subvectors, quantize each with a 256-entry codebook, and store a 512-byte vector in 8 — while m small codebooks span k^m codes. The compression math, asymmetric distance lookups, and IVFPQ — worked math plus runnable code.
GraphRAG
RAG for the global questions vector retrieval can't answer. GraphRAG builds a knowledge graph from the corpus, detects communities of related entities, summarizes each, and map-reduces those summaries — so "what are the main themes across everything?" gets a comprehensive answer. Why top-k retrieval under-covers global queries, and the graph pipeline — worked math plus runnable code.
Textbooks Are All You Need (Phi)
A 1.3B model trained on curated, textbook-quality data rivaled models 10× larger. Data quality, not just scale, drives capability — a small clean dataset can carry more effective signal (tokens × quality) than a giant noisy one, and the small model is far cheaper to run. The effective-signal argument, compute math, and caveats — worked math plus runnable code.
Transformer-XL
The transformer that broke the fixed-context limit. Segment-level recurrence caches the previous segment's hidden states so context grows with depth, and relative positional encoding makes the recurrence coherent. Why vanilla transformers fragment context, how the cache reaches thousands of tokens back, and the ~1,800× evaluation speedup — worked math plus runnable code.
Ring Attention
Context that scales with the number of devices. Shard the sequence across a ring, rotate the key/value blocks around it so every query sees every key, and per-device memory stays O(N/P) — with communication hidden behind computation. Exact dense attention, no approximation, pushing context into the millions. The rotation and overlap math — worked math plus runnable code.
Muon
The optimizer that orthogonalizes weight-matrix updates instead of scaling them element-wise like Adam. Pushing every singular value of the momentum to 1 spreads learning across all directions, done cheaply with Newton-Schulz iteration (matmuls, no SVD). Why it pairs with Adam for 1D params, and how it scaled to trillion-parameter training — worked math plus runnable code.
Flamingo
The visual language model that bridges a frozen vision encoder and a frozen LLM with a small trained connector. A Perceiver Resampler compresses each image into a fixed set of tokens, and gated cross-attention (tanh gate initialized at 0) injects them without breaking the language model — enabling few-shot multimodal in-context learning. The resampler and gating math — worked math plus runnable code.
DINOv2
Label-free visual features by self-distillation: a student learns to match a teacher that is an EMA of itself, with centering and sharpening to prevent collapse. DINOv2 scaled this with curated data into a general vision backbone that works frozen across detection, segmentation, and depth. The self-distillation loop and anti-collapse math — worked math plus runnable code.
3D Gaussian Splatting
Real-time photorealistic radiance fields. Represent a scene as millions of 3D Gaussians and render by projecting and alpha-compositing them front-to-back — fast rasterization instead of NeRF's slow ray-marching, hitting 100+ FPS. The compositing equation (C = Σ cᵢαᵢTᵢ), the differentiable optimization that fits Gaussians to photos, and why explicit beats implicit — worked math plus runnable code.
NeRF
The paper that launched the radiance-field era. Store a 3D scene as a small MLP mapping (position, direction) → (color, density), and render photorealistic novel views by volume rendering along rays. The rendering integral (α = 1 − exp(−σδ), C = Σ Tᵢαᵢcᵢ), why positional encoding unlocks sharp detail, and how it set up Gaussian Splatting — worked math plus runnable code.
LLaVA
Visual instruction tuning the simple way. Connect CLIP to an LLM with a single linear projection, prepend the image tokens to the prompt, and instruction-tune on data a text-only GPT-4 generated — a capable visual assistant on a shoestring. The minimal connector (a matrix, <0.1% of the LLM), the synthetic-data trick, and two-stage training — worked math plus runnable code.
STaR
A model teaching itself to reason. STaR generates chain-of-thought rationales, keeps only the ones that reach the correct answer, fine-tunes on them, and repeats — bootstrapping reasoning from question-answer pairs alone. Rationalization (hinting the answer) rescues hard problems. The self-improvement loop, foreshadowing RL-trained reasoning models — worked math plus runnable code.
AlphaGo
The system that beat a Go world champion by fusing Monte Carlo Tree Search with deep networks: a policy network to propose moves and a value network to judge positions. How PUCT selection balances exploiting good moves against exploring promising ones, the supervised-then-self-play training, and the test-time-search idea it seeded — worked math plus runnable code.
MuZero
Planning without knowing the rules. MuZero learns its own model — representation, dynamics, and prediction networks — and runs tree search entirely in a learned latent space, so it works where no simulator exists (like Atari). The three networks, the value-equivalence idea (a model trained to be useful for planning, not to reconstruct observations), and the n-step return — worked math plus runnable code.
AlphaFold 2
The system that solved the 50-year protein-folding problem — by reading evolution. Residues that touch in a folded protein mutate together across related sequences, so covarying columns of a multiple-sequence alignment reveal 3D contacts; the Evoformer refines them with attention and a structure module predicts coordinates. The coevolution insight, the Evoformer, and pLDDT confidence — worked math plus runnable code.
Codex
The GPT-on-code model behind GitHub Copilot — and the evaluation it standardized. Codex introduced HumanEval and pass@k: grade generated programs by whether they pass the unit tests, and measure the chance at least one of k samples works. The unbiased pass@k estimator (1 − C(n−c,k)/C(n,k)) and why sampling helps — worked math plus runnable code.
SWE-bench & SWE-agent
Can a model do real software engineering? SWE-bench grades models on resolving actual GitHub issues — the patch must apply, make the failing tests pass, and break no passing tests. SWE-agent gives the model an Agent-Computer Interface (search/open/edit/run) built for the model, not humans. The strict resolution rule and why the interface matters — worked math plus runnable code.
AlphaCode
Median-human competitive programming by scale plus selection. AlphaCode samples millions of candidate programs, filters out those that fail the example tests, clusters the survivors by behavior, and submits a few representatives from the largest clusters — consensus as majority vote over code. The sample-filter-cluster pipeline and why it works — worked math plus runnable code.
Llama 3
A herd of open dense transformers trained far past the Chinchilla point. Llama 3 deliberately over-trains its smaller models on ~15 trillion tokens — nearly 100× the compute-optimal count — because a model is trained once but served billions of times, so inference, not training, dominates the lifetime bill. The training-optimal versus inference-optimal trade-off, with worked math and runnable code.
Gemini 1.5
A million-token context window with near-perfect needle-in-a-haystack recall. Gemini 1.5, a sparse mixture-of-experts model, can retrieve a fact hidden anywhere in up to 1M (research: 10M) tokens — because its usable attention span covers the whole window (W ≥ L), where a limited window drops every early needle. Why long context is hard, why it is expensive, and how to test it — worked recall condition plus runnable code.
Sora
Text-to-video by treating video as a diffusion transformer over spacetime patches. Sora compresses a video into a latent volume, cuts it into patches — one token each — and denoises the sequence, so resolution, aspect ratio, and duration are all just different patch counts. Why that unifies any shape into one model, and why patch count (linear in duration, quadratic in resolution) sets the compute — worked math plus runnable code.
V-JEPA
A self-supervised video world model that predicts in representation space, not pixels. V-JEPA masks regions of a video and predicts their features (encoder outputs) rather than reconstructing pixels — so the target drops the unpredictable detail a reconstruction loss is forced to model. Why abstraction beats reconstruction, and how a stop-gradient EMA target encoder avoids representation collapse — worked math plus runnable code.
RT-2
A Vision-Language-Action model that controls a robot by emitting actions as text tokens. RT-2 discretizes each dimension of a continuous action into bins — one vocabulary token per dimension — so a web-pretrained vision-language model co-trains on internet data and robot trajectories and its semantic knowledge transfers to control, generalizing to objects and commands never seen in robot data. Action-as-token, quantization precision, and emergent generalization — worked math plus runnable code.
Genie
A generative interactive environment that turns an image into a frame-by-frame playable world — trained on unlabeled video with no action labels. A latent action model infers a small discrete set of actions from consecutive frames, and a dynamics model turns them into control, so a tiny codebook forces consistent, reusable latent actions to emerge unsupervised. Latent-action inference and controllable generation — worked math plus runnable code.
PaLM
A 540-billion-parameter dense model trained with Pathways that made emergent abilities concrete. Some capabilities stay near chance until scale crosses a threshold, then jump sharply — because a task needing k sequential correct steps scores p^k, so a smooth per-step improvement becomes a sudden task-level breakthrough. Compositional emergence, why harder tasks emerge later, and what Pathways enabled — worked math plus runnable code.
Qwen3
An open model family that unifies a thinking mode (chain-of-thought) and a non-thinking mode (direct answer) in one model, with a controllable thinking budget. Since thinking gains saturate and help hard queries far more than easy ones, gating reasoning on difficulty — skip easy, spend the budget on hard — maximizes accuracy per token, beating always-think and never-think. The test-time-compute trade-off as a dial — worked math plus runnable code.
Dense Passage Retrieval
The dual-encoder method that made dense retrieval beat BM25 for open-domain QA. DPR encodes questions and passages into a shared vector space and matches by dot product; its efficiency trick is in-batch negatives — a batch of B pairs yields a B×B similarity matrix whose diagonal is the positive and whose off-diagonal gives B×(B−1) negatives for free. Why matching on meaning beats keywords, and how the free-negatives trick trains it — worked math plus runnable code.
HNSW
The graph index behind fast approximate nearest-neighbor search — the default in nearly every vector database. HNSW wires points into a navigable small-world graph and searches it greedily (hop to the neighbor closest to the query), organized into a skip-list-style hierarchy of layers whose height grows like ln(N) — so search is O(log N) instead of O(N). Greedy navigation, the exponential level assignment, and why squaring the dataset only doubles the layers — worked math plus runnable code.
GPT-4 Technical Report
The report is famous for what it withholds, but its key methodological claim is predictable scaling: OpenAI predicted GPT-4's final loss from models trained with up to ~1000× less compute by fitting a power law and extrapolating. Why loss falls as a smooth power law you can fit on cheap small runs and extend across a huge compute gap — turning a giant, risky training run into an engineering forecast — and where capability prediction still breaks. Worked math plus runnable code.
OpenAI o1
The reasoning model trained with reinforcement learning to think before it answers — and the discovery of a second scaling axis. For a fixed model, accuracy rises roughly log-linearly with test-time (inference) compute: each ×10 in thinking buys a fixed accuracy step, so reaching a higher accuracy costs a multiplicative jump in compute. Reasoning as search under a budget, the inference-scaling law, and its exponential price — worked math plus runnable code.
VAE
The Variational Autoencoder — how to turn a compressor into a generator. A plain autoencoder's latent space is full of holes you can't sample; the VAE encodes each input to a Gaussian cloud and pulls all the clouds toward a standard-normal prior, making the space smooth and samplable. Trained by maximizing the ELBO (reconstruction minus KL), with the KL in closed form and the reparameterization trick z = μ + σ·ε making sampling differentiable. The foundation of the latent space under latent diffusion — worked math plus runnable code.
PPO
Proximal Policy Optimization — the RL algorithm behind RLHF. Naive policy gradients take steps that collapse the policy; PPO caps the step with a clipped surrogate objective, min(r·A, clip(r, 1−ε, 1+ε)·A), where r = π_new/π_old is the probability ratio and A the advantage. Once the ratio leaves the trust region 1±ε in the helpful direction the objective flattens — TRPO-level stability from a first-order clip, no hard constraint. The optimizer InstructGPT and RLHF pipelines used. Worked math plus runnable code.
DQN
The Deep Q-Network that learned Atari from raw pixels and launched deep reinforcement learning. A neural net approximates the Q-function, trained toward the Bellman target y = r + γ·max Q(s′,a′) — immediate reward plus discounted best future value (just r on a terminal step). Two stabilizers made neural Q-learning trainable: experience replay (random minibatches from a buffer break frame correlation) and a frozen target network (a stationary goal so the net doesn't chase itself). The ancestor of AlphaGo and modern deep RL. Worked math plus runnable code.
Classifier-Free Guidance
The "guidance scale" slider behind every diffusion image tool. A conditional diffusion model predicts the noise twice each step — with the prompt (ε_cond) and without it (ε_uncond) — then extrapolates: ε = (1+w)·ε_cond − w·ε_uncond, equivalently ε_uncond + (1+w)·(ε_cond − ε_uncond). w=0 is plain conditioning, larger w pushes further along the conditional direction for tighter prompt-following (too far → over-saturation). No separate classifier — trained with condition-dropout — and the source of negative prompts. Worked math plus runnable code.
Knowledge Distillation
How to compress a big, accurate "teacher" into a small, fast "student" — by training on the teacher's soft probability distribution, not just the hard label. A one-hot label is a thin signal; the teacher's full distribution encodes "dark knowledge" (a 2 looks a bit like a 7, nothing like a cat). The trick: a temperature-scaled softmax, p_i = exp(z_i/T) / Σ exp(z_j/T), softens the distribution as T grows so those tiny wrong-class probabilities become a strong training signal — without ever changing the top class. The student minimizes KL to the softened teacher plus cross-entropy on the labels. The "Distil" in DistilBERT and a first-line model-compression tool. Worked math plus runnable code.
VQ-VAE
The autoencoder that made the latent DISCRETE — and quietly became the tokenizer behind image generation. The encoder outputs a continuous vector, but it is quantized to the nearest entry in a learned codebook {e_1..e_K}: k = argmin_j ‖z_e − e_j‖², so each latent position becomes one of K discrete tokens. The non-differentiable argmin is trained with a straight-through estimator (copy the decoder gradient back to the encoder), plus a three-part loss: reconstruction + codebook ‖sg[z_e]−e‖² + β·commitment ‖z_e−sg[e]‖². Discrete codes let a Transformer or diffusion prior model images/audio as tokens — the lineage behind VQ-VAE-2, DALL·E, and VQGAN. Worked math plus runnable code.
Graph Neural Networks
How a neural network learns on graphs — molecules, social networks, citation webs — where a convolution has nothing to slide over. The Graph Convolutional Network reduces it to message passing: each node updates itself by mixing its own feature with a degree-normalized average of its neighbors. One layer is H' = σ( H W) with  = D̃^(-1/2)(A+I)D̃^(-1/2) — self-loops (A+I) keep a node's own feature and the symmetric normalization stops high-degree hubs from dominating. Stack layers to reach further; too deep and features over-smooth. The template (gather, aggregate, update) behind GraphSAGE and GAT — and attention is message passing on a fully connected graph. Worked math plus runnable code.
GloVe
Word vectors from GLOBAL co-occurrence statistics — the count-based cousin of word2vec. Build a co-occurrence matrix X (X_ij = how often word j appears in the context of word i), then learn vectors so a dot product recovers the LOG count: w_i·w̃_j + b_i + b̃_j ≈ log(X_ij). The insight: co-occurrence RATIOS (P(solid|ice)/P(solid|steam)) carry meaning, and taking logs turns multiplicative ratios into differences a vector space represents — which is why king−man+woman≈queen works. Fit with weighted least squares, J = Σ f(X_ij)(…−log X_ij)², where f(x)=(x/xmax)^0.75 caps frequent pairs and f(0)=0 skips the empty matrix. The standard pretrained word embedding for years. Worked math plus runnable code.
The Lottery Ticket Hypothesis
The finding that a dense, randomly-initialized network hides a small "winning ticket" — a sparse subnetwork that, trained in isolation FROM THE SAME INITIAL WEIGHTS, matches the full model's accuracy. You find it by iterative magnitude pruning: train, drop the smallest-magnitude weights (a binary mask), RESET survivors to their original init, repeat. The famous twist: randomly reinitializing the same sparse structure trains worse — structure and lucky initialization are entangled. After k rounds at prune fraction p, only (1−p)^k of the weights remain. A new lens on why over-parameterization helps, and the seed of sparse-training research. Worked math plus runnable code.
DDIM
Denoising Diffusion Implicit Models — the trick that made diffusion sampling deterministic and 10–50× faster, using the SAME trained DDPM network (no retraining). Each step predicts the clean image x̂0 = (x_t − √(1−ᾱ_t)·ε)/√ᾱ_t, then re-projects to an earlier step: x_{t−1} = √ᾱ_{t−1}·x̂0 + √(1−ᾱ_{t−1}−σ²)·ε + σ·z. A single knob σ = η·√((1−ᾱ_{t−1})/(1−ᾱ_t))·√(1−ᾱ_t/ᾱ_{t−1}) unifies the two samplers: η=1 recovers stochastic DDPM, η=0 gives σ=0 and the deterministic DDIM that can skip timesteps. Determinism unlocks reproducibility, latent interpolation, and DDIM inversion for editing — and reframed sampling as an ODE. Worked math plus runnable code.
Self-Instruct
How to bootstrap a large instruction-tuning dataset from a language model's OWN generations, starting from just 175 human-written seed tasks. The generate→filter→add loop: sample existing tasks as examples, prompt the model to write new instructions + input/output instances, filter, add survivors back, repeat — 175 seeds bloom into ~52K diverse tasks. The quantitative heart is a diversity filter: keep a new instruction only if its ROUGE-L similarity (longest-common-subsequence overlap) to every existing one is below 0.7, so the pool never collapses into near-duplicates. The seed of Alpaca and the open instruction-tuning wave, and a landmark in synthetic data. Worked math plus runnable code.
mixup
The two-line data augmentation that trains on CONVEX COMBINATIONS of example pairs — blending both inputs and labels: x̃ = λ·x_i + (1−λ)·x_j, ỹ = λ·y_i + (1−λ)·y_j, with λ ~ Beta(α,α). A 70/30 blend of a cat and a dog is trained with the soft label "0.7 cat, 0.3 dog." This "vicinal risk minimization" makes the network behave linearly between training points, smoothing its decision boundary. The payoffs, for near-zero cost and no architecture change: better generalization, honest calibration, resistance to label noise, and adversarial robustness. A default augmentation for ResNets and ViTs (often with CutMix). Worked math plus runnable code.
AlphaZero
The single algorithm that mastered Go, chess, and shogi from ZERO human data — pure self-play, same code for all three. A single deep net outputs a policy prior P(s,a) and a value v(s); Monte Carlo Tree Search uses them to look ahead, selecting moves by the PUCT rule: maximize Q(s,a) + c·P(s,a)·√(ΣN)/(1+N(s,a)) — exploit the mean value Q, explore high-prior under-visited moves via the bonus that fades with visits. Search yields an improved policy (visit counts), self-play yields outcomes z, and both train the net (loss = (z−v)² − π·log p). Generalized AlphaGo into a domain-agnostic recipe; the seed of MuZero and the self-play-plus-search paradigm. Worked math plus runnable code.
wav2vec 2.0
Self-supervised speech: learn from raw, untranscribed audio, then fine-tune on as little as TEN MINUTES of labels. A CNN encodes the waveform into latent frames; a span is masked; a Transformer context net must identify each masked frame's TRUE quantized latent among K distractors via a contrastive InfoNCE loss (cosine similarity / temperature): L = −log[exp(sim(c,q_true)/κ) / Σ exp(sim(c,q̃)/κ)]. Targets are discretized through a learned codebook (product quantization + Gumbel-softmax) so the model discovers phone-like units. Brought BERT-style masked pretraining and CLIP-style contrastive learning to audio; the basis of HuBERT and multilingual XLS-R. Worked math plus runnable code.
DeepSeek-V4
The efficiency sequel to V3, aimed at the real cost of long context. A ~1.6T-parameter MoE (only ~49B active per token) that serves a 1M-token window at roughly a tenth of V3.2's KV cache. The mechanism is one idea on two faces — HYBRID sparse attention: compress each token's keys/values into a small latent (memory per token: ratio = d_c/width, context-invariant), then SELECT only the top-k blocks of size b to score against, freezing the scored-key count at k·b once n>k·b while dense attention keeps paying O(n²). The two discounts multiply: FLOPs ratio = (k·b/n)·(d_c/width). Reasoning is post-trained with ON-POLICY DISTILLATION — the student generates its own trajectories and a stronger teacher corrects those exact outputs — replacing R1's large-scale RL loop with a denser, more stable signal. Worked math plus runnable code.
Kimi K3
The sequel to Kimi K2, and a different bet: attack decode SPEED, not KV memory. A ~2.8T open MoE with a 1M-token context — reported as the largest open model to date — built on KIMI DELTA ATTENTION (KDA), a linear-attention delta rule. Softmax attention re-reads the whole KV cache every decode step (O(n) per token, so generation slows as context grows); KDA folds the past into ONE fixed-size recurrent state S (~d×d) and reads o=S·q in constant time regardless of length. The DELTA rule (S ← S − β·(S·k − v)·kᵀ) makes each write CORRECTIVE — overwrite a key's old value instead of accumulating it — which is what plain linear attention gets wrong (write (k,a) then (k,b): delta reads b, linear reads a+b, smeared). ATTENTION RESIDUALS keep a minority of full-attention layers for the sharp exact recall a fixed state blurs — a hybrid, not a wholesale swap. Worked math plus runnable code.
LLMs Get Lost in Multi-Turn Conversation
The ICLR 2026 Outstanding Paper that measured a failure everyone had felt: give a top model a fully-specified task in ONE prompt (concat) and it shines; split the IDENTICAL requirements across several turns (sharded) and 15 leading LLMs fall apart — ~39% average drop. The decomposition is the punchline: APTITUDE (best-case ability) barely moves, but RELIABILITY craters — the spread between a model's best and worst runs roughly DOUBLES. The mechanism is premature commitment: handed partial info, the model guesses a full answer early, locks it in, and when a later turn contradicts it "gets lost and does not recover." A small decay model captures the shape — concat R=p (flat), sharded R=p·(1−q)^(turns−1) (geometric decay), so longer chats get less reliable and the whole loss lives in a reliability term while the ceiling p is untouched. Fixes: front-load the spec, or consolidate/recap a long chat back into one fresh full-spec message. Worked math plus runnable code.
Engram
The memory half of the long-context problem. Softmax attention answers a query by spreading a fixed probability budget over ALL tokens, so as the haystack grows the needle dilutes and recall degrades — you pay O(n) AND lose accuracy. Engram bolts a KEY-ADDRESSED memory onto the model: write a fact under a key, retrieve it in constant time O(1) regardless of context length, no scan and no dilution (a Python dict IS the abstraction). It is CONDITIONAL — a lightweight gate decides whether a position needs a long-range recall, so local tokens pay nothing and only recall tokens incur one O(1) lookup. Because retrieval is by identity not by out-competing every other token, needle-in-a-haystack recall stops falling with length (reported ~84.2→97). Underpins DeepSeek-V4 (cheap context that stays accurate); same explicit-memory family as Titans and MemGPT, baked into the model not bolted on by a harness. Worked math plus runnable code.
Qwen4-Coder
An open, Apache-2.0 coding model that reportedly hits ~82% on SWE-Verified while running on a MacBook — the first open, Mac-runnable model past 80% there. The enabler is the mixture-of-experts SHAPE, not a bigger model: ~32B total parameters but only ~3B active per token, which splits the two costs everyone conflates. STORAGE is set by total params (all experts must be RAM-resident since the router may pick any) — 32B at 4-bit ≈ 16GB, fits a 24GB Mac (fp16 would be 64GB and fail); quantization is what makes it a laptop model. SPEED is set by ACTIVE params (decode is bandwidth-bound and reads only the routed ~3B experts) — so a 32B model decodes at 3B throughput. The mental model: "does it fit?" is answered by total×bits÷8; "is it fast enough?" by active params vs bandwidth — dense models tie these together, MoE pulls them apart. Sequel to Qwen3; validates the local-first coding-agent stack. Worked math plus runnable code.
GPT-5.6 System Card
A modern system card is really a COMPUTE-ALLOCATION POLICY, not one model. TIERED ROUTING (reported Sol/Terra/Luna) sorts each query by difficulty — cheap fast tier by default, escalate the hard cases (the cascade cost math lives in the model-routing handbook — linked, not re-derived). ULTRA MODE spends test-time compute in PARALLEL: run several independent agents (reported 4) on the hardest task and a VERIFIER keeps any that solves, so the solve rate is 1−(1−p)^N. The fresh angle = why FOUR, not forty: the marginal value of the N-th agent is Δ(N)=(1−p)^(N−1)·p — geometric decay (p=0.6: +0.60/+0.24/+0.10/+0.04) while cost is linear in N, so there's a knee. Runnable proves best-of-N rises, marginal=(1−p)^(N−1)·p, diminishing returns, 4th≪1st, value/cost decreasing, and finds the knee. PREPAREDNESS: safety case must cover the STRONGEST config (top tier + full ultra), not the cheap default. Cross-links o1-system-card (scaling curve) + model-routing/ai-cost-engineering + self-consistency. Worked math plus runnable code.
Concept Circuits
The foundation of mechanistic interpretability — how to read a model. Neurons are POLYSEMANTIC (one unit fires for green AND legal text AND Python) because the model stores concepts as DIRECTIONS spread across many neurons, not one-per-neuron. SUPERPOSITION: a d-dim space holds only d orthogonal directions but VASTLY more near-orthogonal ones, so a model packs in more features than dimensions and accepts small interference (interference(i,j)=feature_i·feature_j: orthogonal=0, superposed small-nonzero; 3 unit vecs at 120° in 2-D → pairwise dot −0.5). It works because features are SPARSE — few active at once, so collisions rarely bite and a lone active feature reads back cleanly. ATTRIBUTION GRAPHS trace which features caused an output (contribution = output-direction · feature-direction) and chain them across layers into a CIRCUIT. Runnable proves orthonormal self=1/cross=0, 3 features in 2-D, unit length preserved, interference bounded 0.5, clean sparse readout. First interp content sitewide; pairs the coming mech-interp-for-engineers handbook. Worked math plus runnable code.
The 95% Number, Examined
The most-quoted statistic in enterprise AI, read carefully: roughly 95% of pilots reportedly produced no MEASURABLE profit-and-loss impact. Two words carry the sentence. "Measurable" means attributable financial effect — not accuracy, not satisfaction — and "pilot" means a bounded trial, which selects for projects that end before an accounting period closes. Three separate failures hide inside the one number: MEASUREMENT failure (it worked and no baseline exists, because nobody captured the before-picture in week one and it is unrecoverable afterwards), ATTRIBUTION failure (something improved and three other things changed the same quarter, so finance will not credit yours), and ACTUAL failure (the workflow was wrong or nobody adopted it). The first two are why the forward deployed role exists — both are solved by work before and after the model. Sourced, confidence-labelled (reported via press coverage, not a public methodology), and paired with the a16z argument it mirrors.
Trading Margin for Moat
Why a software company would deliberately hire expensive engineers to do customer work. The services-led-growth thesis: spend gross margin on deployment depth because the resulting integration is hard to displace and the outcome is provable — the margin hit is an acquisition cost for defensibility, not an inefficiency. The arithmetic that decides whether it holds, as a worked napkin example: an FDE at ~$300K loaded doing 3 deployments a year costs ~$100K per deployment, which is 67% of a $150K ACV and 17% of a $600K one — and REUSE is the only term that improves over time (0.6x, then 0.4x). The failure mode built into the thesis is paying the margin and not receiving the moat, which arrives one reasonable exception at a time. Diagnostic: does deployment N take measurably less time than N−1? Plus what the trade means for your career and the two questions to ask an employer.
1,000 FDE Jobs, Analysed
Reading a thousand job postings beats reading a thousand opinions. One title covers at least THREE distinct jobs — builder (few customers, deep, milestone-driven), pre-sales (many customers, shallow, quarter-driven, often with quota), and internal/platform — and roughly a third of postings using the FDE title are the pre-sales shape, which the posting rarely makes explicit. A separate census counted 1,206 strict-definition postings across 669 companies at a ~$185K median posted base. What every posting asks for: strong general engineering, LLM app patterns, integration reality, deployment where you do not own the cloud, and increasingly MCP servers as deliverables — with EVALUATION the most-cited hard skill and the least taught anywhere. How to read a posting properly, the two questions that settle which species it is, and the honest caveat that posting counts measure demand signals rather than filled roles.
Paper Breakdowns — frequently asked questions
Are the paper breakdowns free to watch?
Yes. Every breakdown is a free narrated video plus a written companion — no paywall, no sign-up.
Who makes these breakdowns?
They are written and narrated by Saurabh Singh, a senior AI engineer, focused on making foundational research actually intuitive — real analogies and animated visuals, not just equations read aloud.
Do I need a research background to follow along?
No. Each breakdown is built for engineers and curious builders, not academics — every idea is explained with an everyday analogy before the formal version, if there is one.
Which paper should I start with?
"Attention Is All You Need" — it introduced the Transformer, the architecture underneath GPT, BERT, Claude, and Gemini. Nearly everything else in modern AI builds on it.
A paper missing?
If there's a paper you'd like broken down, send it through. Genuine requests shape what gets covered next.