Textbooks are
all you need.

The scaling laws gave the field a simple, powerful recipe: capability rises predictably with model size, data, and compute. Bigger, trained on more, is better. It worked so well it became the default strategy — pour in web-scale data, grow the parameter count, watch benchmarks climb.

But that framing quietly assumes all tokens are equal. Web data is not: it's a mix of the brilliant and the banal, the correct and the wrong, the novel and the thousandth near-duplicate. Phi's question was whether the composition of the data — its quality — is a lever as powerful as its sheer quantity. If you could feed a model only the good stuff, how small could the model, and the dataset, be?

Phi's core move is to treat data quality as the primary design variable. It built a training set of what the authors call textbook-quality data — clear, correct, self-contained, pedagogically structured examples, the way a good textbook explains a concept rather than the way a random forum post does. Two ingredients:

The formalization is simple. Not every token carries useful learning signal; if a fraction q (the quality) does, the effective signal in a dataset of N tokens is N·q. Web data is enormous N with tiny q; textbook data is small N with q near 1 — and the product can favor the small clean set.

Forty times fewer raw tokens, more actual teaching — because the curated set wastes almost nothing. The runnable version below computes this trade and the compute a smaller model saves.

The result was striking. phi-1, at 1.3 billion parameters trained on this small curated set, matched or beat models roughly an order of magnitude larger on Python coding benchmarks like HumanEval. The follow-ups (phi-1.5, phi-2, and the phi-3 family) extended the recipe beyond code to general reasoning, repeatedly showing small models punching far above their parameter count.

And the win compounds at deployment. A forward pass costs roughly 2 × params FLOPs per token, so a 1.3B model is about ten times cheaper per token than a 13B one. A small model that matches a big one on quality doesn't just train cheaply — it runs cheaply forever after, which for anything served at scale is the number that matters most.

Why should clean data be so much more efficient? A few reasons stack up. Redundancy: web corpora repeat the same easy patterns endlessly, so most tokens teach nothing new — the model has already learned them. Noise: incorrect or low-quality examples actively mislead, spending capacity on things you'd rather the model not learn. Density: a textbook introduces concepts deliberately, in a logical order, with worked examples — a far higher rate of genuinely new, correct information per token.

Put together, curated data means the model spends every parameter and every gradient step on signal instead of drowning it in noise and repetition. That's why a small model can keep up: it isn't wasting its limited capacity memorizing junk. Quality doesn't add a constant bonus — it changes the exchange rate between tokens and learning.

The result was influential and also debated. Two honest cautions travel with it:

None of these overturned the central finding — that data quality is a first-class lever — but they are why the community treated the specific benchmark numbers with care while embracing the broader thesis. Measuring quality, and guarding against contamination, became active areas precisely because phi made quality matter.

Phi helped shift the field's attention from "scale everything" to "curate ruthlessly." The small language model movement — capable models in the 1–4B range that run on a laptop or a phone — owes much to the demonstration that careful data can substitute for raw size. Data curation, filtering, and synthetic generation became central research topics, not afterthoughts, and "data-centric" work gained the status the scaling laws had given model-centric work.

Its lasting idea complements, rather than contradicts, scaling. The scaling laws describe how capability grows with size for a fixed data quality; phi showed that quality is another axis entirely, one you can push to buy the same capability at a fraction of the size and cost. Both are true — and the practical frontier is to scale and curate, spending compute where it teaches most.