Textbooks are
all you need.

Rung 5 answers "how many tokens" and is silent on "which tokens". Its curves are fitted over a fixed corpus, so data is a scalar — a count. This rung's claim is that the count is not the whole story: that curating and synthesising higher-quality training data changes the loss-versus-compute picture enough to matter, and it argues the case in the most awkward possible way — by training a model far off Chinchilla-optimal and doing well anyway.

In the ladder's units, this is the smallest model on it since rung 1: 1.3B parameters, trained on about 7B unique tokens of filtered and synthetically generated "textbook-quality" material, seen about 50B times over. That is only about 5 unique tokens per parameter (about 38 seen, with heavy repetition), far below rung 5's ~20:1 in unique data — a configuration rung 5 would call data-starved — and it reports roughly 50% pass@1 on HumanEval, in the neighbourhood of models an order of magnitude larger.

Read against rung 5, the two are not in conflict so much as orthogonal: Chinchilla tells you how to spend a budget on a given corpus, and this rung says the corpus is a variable you were treating as a constant.

The scaling laws gave the field a simple, powerful recipe: capability rises predictably with model size, data, and compute. Bigger, trained on more, is better. It worked so well it became the default strategy — pour in web-scale data, grow the parameter count, watch benchmarks climb.

But that framing quietly assumes all tokens are equal. Web data is not: it's a mix of the brilliant and the banal, the correct and the wrong, the novel and the thousandth near-duplicate. Phi's question was whether the composition of the data — its quality — is a lever as powerful as its sheer quantity. If you could feed a model only the good stuff, how small could the model, and the dataset, be?

Phi's core move is to treat data quality as the primary design variable. It built a training set of what the authors call textbook-quality data — clear, correct, self-contained, pedagogically structured examples, the way a good textbook explains a concept rather than the way a random forum post does. Two ingredients:

The formalization is simple. Not every token carries useful learning signal; if a fraction q (the quality) does, the effective signal in a dataset of N tokens is N·q. Web data is enormous N with tiny q; textbook data is small N with q near 1 — and the product can favor the small clean set.

Forty times fewer raw tokens, more actual teaching — because the curated set wastes almost nothing. The runnable version below computes this trade and the compute a smaller model saves.

Why should clean data be so much more efficient? A few reasons stack up. Redundancy: web corpora repeat the same easy patterns endlessly, so most tokens teach nothing new — the model has already learned them. Noise: incorrect or low-quality examples actively mislead, spending capacity on things you'd rather the model not learn. Density: a textbook introduces concepts deliberately, in a logical order, with worked examples — a far higher rate of genuinely new, correct information per token.

Put together, curated data means the model spends every parameter and every gradient step on signal instead of drowning it in noise and repetition. That's why a small model can keep up: it isn't wasting its limited capacity memorizing junk. Quality doesn't add a constant bonus — it changes the exchange rate between tokens and learning.

The model in "Textbooks Are All You Need" is ordinary; the method is the data pipeline, and two stronger OpenAI models do most of the work in it. The paper's thesis is that capability depends less on model size than on how instructive each training token is, and this pipeline is its working definition of instructive. In order:

1. Start from deduplicated public code. The Python subset of The Stack plus StackOverflow: about 35M files and samples, more than 35B tokens.
2. Label a small sample with GPT-4. About 100k samples were shown to GPT-4 with a prompt asking it to judge each snippet's educational value for a student whose goal is to learn basic coding concepts.
3. Train a cheap filter and run it everywhere. A random-forest classifier learns to predict GPT-4's judgment from the snippet's output embedding under a pretrained CodeGen model. Running that classifier, not GPT-4, over the full corpus is what makes filtering 35B tokens affordable. About 6B tokens survive.
4. Generate synthetic textbooks. GPT-3.5 writes under 1B tokens of Python textbook text, explanations interleaved with code. To stop the samples repeating each other, generation is steered by constraints on topic and target audience; the authors point out that diversity, not just correctness, is the hard part of synthetic data.
5. Pre-train phi-1-base on CodeTextbook. Filtered code plus synthetic textbooks, about 7B tokens. The model is a 1.3B-parameter decoder-only Transformer: 24 layers, hidden size 2,048, 32 attention heads of size 64, rotary position embeddings and FlashAttention, with attention and MLP blocks run in parallel as in CodeGen and GPT-J, and CodeGen's tokenizer. Training uses AdamW with an effective batch of 1,024 sequences of 2,048 tokens. The released base checkpoint is the one at 24,000 steps: about eight passes over the data and a little over 50B tokens seen. It scores 29% on HumanEval.
6. Generate CodeExercises. GPT-3.5 writes about 180M tokens of exercises, each a function signature and docstring to complete, followed by a solution. Diversity again comes from constraints, this time on the function names in the prompts.
7. Fine-tune into phi-1. Continue training phi-1-base on CodeExercises with the same next-token loss in a short run at a lower learning rate, keeping the best checkpoint. HumanEval pass@1 rises from 29% to 50.6%, and MBPP reaches 55.5%. The paper puts the whole training at about four days on eight A100 GPUs.
8. Repeat at 350M. phi-1-small goes through the same pipeline and reaches 45% on HumanEval.

Two claims the recipe makes

First, that filtering and synthesis change what each token is worth: the paper trains the same model sizes on the unfiltered Stack and StackOverflow data and scores markedly lower on HumanEval. Second, that a small fine-tune can unlock more than it contains. After CodeExercises, phi-1 became noticeably better at things the exercises never covered, such as using libraries like Pygame and Tkinter, which the authors read as fine-tuning reorganizing knowledge learned in pre-training rather than adding new knowledge.

• φCodeGen(x) — the embedding of snippet x from a pretrained CodeGen model; f — the random forest; y^GPT-4 — GPT-4's educational-value judgment on the labeled sample.
• L — ordinary next-token cross-entropy. Nothing in the objective is new; the novelty is entirely in which x get sampled.
• Dseen — tokens seen, not unique tokens: about 50B seen from about 7B unique, roughly 38 per parameter. That is above Chinchilla's ~20, even though the unique-token ratio is only about 5.
• C — about 3.9 × 10^²⁰ FLOPs for N = 1.3 × 10^⁹, roughly 800 times less than GPT-3's 3.14 × 10^²³.

At the extremes: a classifier that keeps everything reproduces the unfiltered baseline, which the paper shows scoring markedly lower; one that keeps almost nothing leaves too few unique tokens, so the same data repeats more and more often within a fixed budget.

Evaluation is pass@1 on HumanEval and MBPP, Python function-completion benchmarks checked by unit tests, against code models from a few hundred million to hundreds of billions of parameters.

phi-1 matched or beat models roughly an order of magnitude larger, though not all of them: WizardCoder, about 12 times larger, still scored higher on HumanEval in the paper's own comparison. The gain compounds at deployment, because a forward pass costs about 2 FLOPs per parameter per token, so a 1.3B model is about ten times cheaper per token than a 13B one. The follow-ups, phi-1.5, phi-2 and the phi-3 family, carried the recipe from code to general reasoning.

Ablations and checks

Training the same model sizes on the unfiltered Stack and StackOverflow data scored markedly lower, and the CodeExercises fine-tune was the largest single step, from 29% to 50.6%. Because those exercises resemble HumanEval, the authors searched them for problems similar to the benchmark using n-gram overlap and embedding and syntax-tree similarity, pruned everything flagged, more than 40% of CodeExercises at the strictest setting, and retrained: the pruned model still outperformed StarCoder. They also wrote new problems designed to be unlike anything online and graded solutions with GPT-4. The paper notes phi-1's limits plainly: it is specialized to Python, and less robust than larger models to stylistic variation or errors in the prompt.

The paper is upfront that its model is narrow: it is trained for Python code and is not a general-purpose model, so the headline benchmark result should not be read as general capability. It also notes the obvious dependency — a large part of the training data is generated by a stronger model, so the recipe presupposes access to one.

The serious external objection is contamination, and it should be stated plainly because it goes to the heart of the claim: when a substantial fraction of training data is synthesised by a model that has itself seen benchmark-adjacent material, a benchmark score is a weaker measurement than usual. Independent analyses raised this about this line of work, the authors have engaged with it, and it has not been settled to everyone's satisfaction. The right posture for a reader is that the direction is well supported and the specific benchmark margin is not the part to quote.

How it held up: data curation and synthetic data became standard, which is the durable result — every frontier training mix since is a designed artefact rather than a scrape, and small models trained on heavily curated data are now a normal product category rather than a curiosity. What remains genuinely open is how far the effect goes, and whether benchmark numbers on synthetically-trained models measure what they appear to.

The result was influential and also debated. Two honest cautions travel with it:

None of these overturned the central finding — that data quality is a first-class lever — but they are why the community treated the specific benchmark numbers with care while embracing the broader thesis. Measuring quality, and guarding against contamination, became active areas precisely because phi made quality matter.

Phi helped shift the field's attention from "scale everything" to "curate ruthlessly." The small language model movement — capable models in the 1–4B range that run on a laptop or a phone — owes much to the demonstration that careful data can substitute for raw size. Data curation, filtering, and synthetic generation became central research topics, not afterthoughts, and "data-centric" work gained the status the scaling laws had given model-centric work.

Its lasting idea complements, rather than contradicts, scaling. The scaling laws describe how capability grows with size for a fixed data quality; phi showed that quality is another axis entirely, one you can push to buy the same capability at a fraction of the size and cost. Both are true — and the practical frontier is to scale and curate, spending compute where it teaches most.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →