Knowledge
distillation.

Rung 1, so the comparison is against training a small model directly on the labelled data. That is the obvious approach and it underperforms, which is the puzzle this rung explains: the small model has enough capacity to represent a good function and cannot find it from labels alone.

The change is to train it on the large model's output distribution instead of on the labels. A one-hot label says "this is a 3" and nothing else. A trained teacher says "very probably a 3, somewhat like an 8, nothing like a cat", and that relative structure over the wrong answers carries a great deal about how the teacher has organised the problem — the similarity geometry it learned, which the label does not contain. One example paired with a full distribution is worth many examples paired with labels.

The mechanism that makes it usable is a detail worth knowing because it is routinely got wrong. A confident teacher's distribution is nearly one-hot, so the informative small probabilities are numerically invisible. Dividing the logits by a temperature before the softmax flattens the distribution and brings them into view, and the same temperature is applied to the student so the two are compared on equal terms. Because that softening also scales the gradients down, they are multiplied back by the square of the temperature to keep the loss's magnitude comparable to the ordinary one it is usually mixed with.

Big models are accurate and expensive; small models are cheap and worse. The obvious way to make a small model is to train it on the same labeled data — but a hard label is a thin signal. "This image is a 2" is a single bit of guidance per example: right class, nothing else. A large, well-trained teacher model, by contrast, outputs a full distribution — maybe 99% "2", 0.9% "7", 0.05% "3", and a whisper of everything else. That distribution encodes a rich similarity structure the model discovered: which classes look alike, which never co-occur.

Knowledge distillation (Hinton, Vinyals & Dean, 2015) trains the small student to match the teacher's soft output distribution rather than only the hard label. That richer target carries far more information per example, so the student can reach accuracy much closer to the teacher than hard-label training allows — a way to compress the teacher's competence into a fraction of the parameters. The one subtlety is that the useful information lives in tiny probabilities that are usually invisible. Fixing that is the paper's core trick, and it's one number: temperature.

The valuable part of the teacher's output isn't the top class — the label already tells you that. It's the relative probabilities of the wrong classes. Hinton called this dark knowledge: the network's learned sense that a "2" is more 7-like than 3-like and utterly unlike a "cat." That structure is a distilled summary of everything the big model learned about how the world's categories relate, and a student that absorbs it generalizes better than one told only the answer.

The problem: for a confident, well-trained teacher, those wrong-class probabilities are minuscule — 0.9% and 0.05% and 0.001% — so close to zero that they contribute almost nothing to a standard training loss, which is dominated by the near-1.0 top class. The dark knowledge is there, but drowned out. You need a way to amplify the small probabilities so the student actually learns from them, without changing which class is on top. That's exactly what a temperature does to a softmax.

A softmax turns raw scores (logits) into probabilities. Adding a temperature T — dividing every logit by T before the softmax — controls how sharp or soft the result is:

Raising T spreads probability mass off the top class and onto the others, proportionally to what the teacher already believed — so the 7 gets a meaningful share and the cat stays near zero. The distribution's entropy rises (it's "softer"), and the faint dark knowledge becomes a loud training signal. Crucially, temperature scaling is monotonic: it changes the magnitudes but never the ordering, so the most-likely class stays most-likely. During distillation, both teacher and student produce their distributions at the same high T, and the student is trained to match the teacher's softened targets; at inference the student drops back to T = 1. One scalar turns a hidden similarity structure into something a small model can be taught.

The temperature-scaled softmax divides each logit by T before exponentiating:

Logits [2, 1, 0]: at T=1 → [0.665, 0.245, 0.09] (sharp); at T=5 → [0.402, 0.329, 0.269] (soft). Higher T raises entropy but keeps class 0 on top.

The student minimizes the cross-entropy between the teacher's softened distribution and its own — equivalently KL(teacher ‖ student), since the teacher's entropy is a constant — usually alongside the standard cross-entropy on the true labels. The soft term is multiplied by T² so its gradients keep the same scale as the temperature changes:

The KL term transfers the teacher's dark knowledge; the cross-entropy term keeps the student grounded in the truth. When the student matches the teacher exactly, the KL is 0 — a perfect distillation of that example. The runnable version below computes the softened softmax, its entropy, and the KL.

The recipe has two halves: make the teacher's output informative, then train the student on it without throwing away the true labels. Here is the procedure as the paper ran it.

1. Train a strong teacher. The teacher is a large model or an ensemble, trained the normal way with as much regularization as it needs. On MNIST it was a net with two hidden layers of 1,200 rectified linear units, trained on all 60,000 training images with dropout and small random shifts of the input. Its only job afterwards is to produce good output distributions.
2. Choose a transfer set. The student learns from the teacher's predictions on a transfer set: the original training set or a pool of unlabeled inputs, because soft targets need no labels.
3. Soften both models with one temperature. For each transfer example the teacher's logits are divided by a temperature T before the softmax, producing soft targets. The student's logits are divided by the same T, so the two distributions are compared at the same sharpness.
4. Soft loss. The first objective is the cross-entropy between the teacher's soft targets and the student's softened predictions.
5. Hard loss. When labels exist, the second objective is ordinary cross-entropy with the correct label, computed from the same student logits at T = 1.
6. Weight and rescale. The two terms are combined as a weighted average. The soft term is multiplied by T², because its gradients shrink roughly as 1/T²; the rescaling keeps the balance between the terms about the same when T is changed. The best results came from a considerably lower weight on the hard term.
7. Deploy at T = 1. The temperature exists only during training. The finished student runs with an ordinary softmax.

What the temperature does to the gradient

For the soft term, the gradient with respect to a student logit z_i is (q_i − p_i)/T, where q is the student's softened distribution and p the teacher's. When T is large compared with the logits, both softmaxes become nearly linear, and if each example's logits are shifted to zero mean the gradient becomes (z_i − v_i)/(N T²), with v the teacher's logits and N the number of classes. At high temperature, distillation is therefore matching the logits themselves, the earlier compression method of Caruana and colleagues, which the paper presents as a special case of its own.

At lower temperatures the student pays less attention to logits far below the average. The paper argues this can help: those very negative logits are barely constrained by the teacher's own training and may be noisy, so a student too small to match everything does better ignoring them. That is why the best temperature depends on the size of the student.

The third experiment: specialists

On JFT, an internal Google dataset of 100 million labeled images in 15,000 classes, the authors also trained 61 specialist models, each on a group of classes the generalist tended to confuse, plus one catch-all class for everything else. Each specialist started from the generalist's weights, and at test time the specialists covering the generalist's top classes refine its prediction. This part is an ensemble method rather than distillation; distilling the specialists back into one model was left as future work.

• v, z — teacher and student logits; p, q — their softmaxes at temperature T; N — the number of classes.
• H(p, q) — cross-entropy against the teacher's soft targets. The teacher's entropy H(p) does not depend on the student, so minimizing it is the same as minimizing KL(teacher ‖ student), not KL(student ‖ teacher).
• T² — cancels the 1/T² shrinkage, so the soft gradient keeps its size as T changes.
• λ — the weight on the true label y. The symbol is ours; the paper describes a weighted average and found a small hard-label weight worked best.

At the extremes: with T = 1 and a confident teacher, p is nearly one-hot and the soft term adds little beyond the label. As T grows, T² times the soft gradient tends to (z_i − v_i)/N, so the objective becomes squared-error matching of logits. With λ = 1 it is ordinary supervised training.

The paper ran three studies: MNIST digits, a production-scale speech recognizer, and the JFT image dataset.

The speech model had about 85 million parameters and was trained on roughly 2,000 hours of English, around 700 million training frames; WER is word error rate. The distilled single model kept more than 80 percent of the frame-accuracy gain the ten-model ensemble had over the baseline, and matched its word error rate. The 3 percent result is the paper's clearest evidence that soft targets act as a regularizer: with labels alone the small-data model overfit badly, while the same data with soft targets from the full model came within two points of the full-data baseline without early stopping.

Two MNIST ablations show where the limits are. With every 3 removed from the transfer set, the student made 206 test errors, 133 of them on the 1,010 test threes; raising the bias of the 3 output by 3.5 cut that to 109 errors, 14 of them threes, so the student had learned what a 3 looks like purely from soft targets on other digits. And temperature mattered most for a tiny student: at 300 or more units per layer any T above 8 worked about equally well, but at 30 units per layer T between 2.5 and 4 was clearly better than higher or lower values. On JFT, the 61 specialists raised top-1 accuracy from 25.0 to 26.1 percent, a 4.4 percent relative gain.

The method's preconditions are acknowledged: it needs a teacher worth copying and a student with enough capacity to copy it, and transfer works best when the student sees a large volume of data through the teacher — unlabelled data is fine, which is a substantial practical advantage that the paper notes.

The mechanism remains genuinely contested, and the disagreement is interesting rather than academic. Candidate explanations include the similarity structure over wrong classes as described, a form of input-dependent label smoothing that regularises, and the teacher's distribution acting as a per-example difficulty weighting. Careful studies have also found that students frequently do not match the teacher's function closely even while improving in accuracy — so whatever is transferring, "the student learns the teacher's function" is not a safe description of it.

How it held up: it is the standard way a deployable small model gets made, and its reach extends well past compression. Every rung of the training ladder's pruning story ends up here because structured compression beats sparsity on real hardware; the strongest small reasoning models on the reasoning ladder were distilled from a large one rather than trained with reinforcement learning directly; and on-policy distillation is this idea moved into post-training. A 2015 compression paper is load-bearing for how frontier capability propagates downward.

Knowledge distillation became one of the standard levers for building efficient models. When you need something that runs on a phone, at the edge, or cheaply at scale, distilling a big teacher into a small student is a first-line technique. It's the "Distil" in DistilBERT and the idea behind TinyBERT and a long line of compact, deployable models that retain most of a large model's accuracy at a fraction of the cost — pairing naturally with other efficiency methods like low-rank adaptation and quantization.

The idea also generalized well beyond its original classification setting. Modern small language models are frequently trained on text generated by much larger models — a form of distillation where the teacher's outputs (not its probability vectors but its actual completions) become the student's training data. Microsoft's Phi line leaned on carefully curated and model-generated "textbook-quality" data to make small models punch far above their weight, and "distill a strong teacher into a small student" is now a common recipe across the efficient-LLM landscape. The 2015 insight — that a model's soft outputs carry knowledge the label throws away — turned out to be one of the most quietly influential ideas in making capable AI cheap enough to deploy.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Five papers about the bill. Each one buys throughput back from a different part of the stack: the weights, the memory hierarchy, the KV cache, the decode loop.

1. 1Knowledge Distillation2014
2. 2FlashAttention2022
3. 3PagedAttention2023
4. 4Speculative Decoding2023
5. 5Medusa2024

All 110 papers, in 19 ladders →