What is 'dark knowledge' and the temperature?

Dark knowledge is the information hidden in the relative probabilities the teacher assigns to the wrong answers. A well-trained classifier might be 99% sure an image is a '2', but the tiny leftover probability reveals that it thinks the '2' looks a bit like a '7' and almost nothing like a 'cat' — a similarity structure that's genuinely useful to learn. Those probabilities are usually too close to zero to matter, so distillation applies a temperature to the softmax: dividing the logits by a temperature T > 1 softens the distribution, amplifying the small probabilities so the dark knowledge becomes a strong training signal. Higher temperature means a softer, higher-entropy target that exposes more of this structure.

How is the student trained?

The student is trained to match the teacher's temperature-softened soft targets, typically by minimizing the KL divergence between the student's and teacher's softened distributions, usually combined with the ordinary cross-entropy loss on the true hard labels. Both the teacher and the student use the same temperature when producing the soft distributions during training; at inference the student uses temperature 1. The soft-target term transfers the teacher's dark knowledge, and the hard-label term keeps the student grounded in the ground truth. Balancing the two lets the small student learn both what's correct and the teacher's nuanced sense of how classes relate.

Why does knowledge distillation matter today?

Distillation is one of the main ways to get small, cheap, fast models that punch above their size — essential for deploying on phones, edge devices, or at scale where inference cost dominates. It underlies compact models like DistilBERT and TinyBERT, and the idea generalizes far beyond the original classification setting: modern small language models are often trained on outputs generated by much larger models (a form of distillation), and 'distilling' a capable teacher's behavior into a smaller student is a common recipe for efficient LLMs. The core insight — that a model's soft outputs carry transferable knowledge beyond the label — has proven remarkably durable.

Paper Breakdowns / Knowledge Distillation

Paper 92~11 min read2015worked math + runnable code

Paper Breakdown

Knowledge
distillation.

Q: What is knowledge distillation?

Knowledge distillation (Hinton, Vinyals & Dean, 2015) is a technique for compressing a large, accurate 'teacher' model into a small, fast 'student' model by training the student to mimic the teacher's output probabilities rather than just the correct label. The key idea is that a teacher's full probability distribution over classes carries far more information than a one-hot label: it says not only which class is right, but how the wrong classes relate to each other. Training the student to match this richer signal lets a small model reach accuracy much closer to the big one than training on hard labels alone, which is why distillation is a standard tool for deploying efficient models.

A one-hot label tells a model almost nothing: "this is a 2," full stop. But a well-trained network knows much more — that this particular 2 looks a little like a 7, and nothing at all like a cat. That extra sense of how the wrong answers relate is real, learnable knowledge, and it's thrown away the moment you collapse it to a single label. Knowledge distillation is the idea of not throwing it away: train a small student to copy a big teacher's full probability distribution, softened just enough to see the faint structure. The result — a tiny model that thinks almost like a huge one. Here's the temperature trick that makes it work.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

More than a label Dark knowledge Turn up the temperature The softened softmax Run it yourself What it showed The compression tool

More than a label

Big models are accurate and expensive; small models are cheap and worse. The obvious way to make a small model is to train it on the same labeled data — but a hard label is a thin signal. "This image is a 2" is a single bit of guidance per example: right class, nothing else. A large, well-trained teacher model, by contrast, outputs a full distribution — maybe 99% "2", 0.9% "7", 0.05% "3", and a whisper of everything else. That distribution encodes a rich similarity structure the model discovered: which classes look alike, which never co-occur.

Knowledge distillation (Hinton, Vinyals & Dean, 2015) trains the small student to match the teacher's soft output distribution rather than only the hard label. That richer target carries far more information per example, so the student can reach accuracy much closer to the teacher than hard-label training allows — a way to compress the teacher's competence into a fraction of the parameters. The one subtlety is that the useful information lives in tiny probabilities that are usually invisible. Fixing that is the paper's core trick, and it's one number: temperature.

The one-sentence version

Train the small student to match the big teacher's probability distribution, softened by a temperature T in the softmax (p_i = softmax(z_i / T)), so the "dark knowledge" in the teacher's small wrong-class probabilities becomes a strong, transferable training signal.

Dark knowledge

The valuable part of the teacher's output isn't the top class — the label already tells you that. It's the relative probabilities of the wrong classes. Hinton called this dark knowledge: the network's learned sense that a "2" is more 7-like than 3-like and utterly unlike a "cat." That structure is a distilled summary of everything the big model learned about how the world's categories relate, and a student that absorbs it generalizes better than one told only the answer.

The problem: for a confident, well-trained teacher, those wrong-class probabilities are minuscule — 0.9% and 0.05% and 0.001% — so close to zero that they contribute almost nothing to a standard training loss, which is dominated by the near-1.0 top class. The dark knowledge is there, but drowned out. You need a way to amplify the small probabilities so the student actually learns from them, without changing which class is on top. That's exactly what a temperature does to a softmax.

Turn up the temperature

A softmax turns raw scores (logits) into probabilities. Adding a temperature T — dividing every logit by T before the softmax — controls how sharp or soft the result is:

How temperature reshapes a distribution

T = 1Ordinary softmax — sharp, top class dominates, dark knowledge invisible.

T > 1Softer — small wrong-class probabilities amplified, dark knowledge exposed.

T → ∞Approaches uniform — every class equally likely.

argmaxThe top class is unchanged by T — softening reveals structure, doesn't flip the answer.

Raising T spreads probability mass off the top class and onto the others, proportionally to what the teacher already believed — so the 7 gets a meaningful share and the cat stays near zero. The distribution's entropy rises (it's "softer"), and the faint dark knowledge becomes a loud training signal. Crucially, temperature scaling is monotonic: it changes the magnitudes but never the ordering, so the most-likely class stays most-likely. During distillation, both teacher and student produce their distributions at the same high T, and the student is trained to match the teacher's softened targets; at inference the student drops back to T = 1. One scalar turns a hidden similarity structure into something a small model can be taught.

The softened softmax

The temperature-scaled softmax divides each logit by T before exponentiating:

p_i = exp(z_i / T) / Σ_j exp(z_j / T) T = 1: normal softmax · T → ∞: uniform

Logits [2, 1, 0]: at T=1 → [0.665, 0.245, 0.09] (sharp); at T=5 → [0.402, 0.329, 0.269] (soft). Higher T raises entropy but keeps class 0 on top.

The student minimizes the KL divergence between its softened distribution and the teacher's, usually alongside the standard cross-entropy on the true labels:

L = KL( p^student_T ‖ p^teacher_T ) + λ · CE( student, hard label ) where KL(p‖p) = 0

The KL term transfers the teacher's dark knowledge; the cross-entropy term keeps the student grounded in the truth. When the student matches the teacher exactly, the KL is 0 — a perfect distillation of that example. The runnable version below computes the softened softmax, its entropy, and the KL.

RUN IT YOURSELF

Soft targets, from scratch

Knowledge distillation hinges on the temperature-scaled softmax: p_i = exp(z_i/T) / Σ exp(z_j/T). It's always a probability distribution (sums to 1). Raising T softens it — the entropy goes up as small wrong-class probabilities get amplified (the "dark knowledge") — and as T grows large it approaches uniform, but the top class never changes. The student is trained to match the teacher's softened distribution; a perfect match has KL divergence 0. Change the logits and the temperature and watch the distribution soften.

CPython · WebAssembly

# Knowledge Distillation (Hinton et al., 2015). Train a small STUDENT to match a big TEACHER's
# SOFT distribution, not just the hard label. A TEMPERATURE T in the softmax, p_i = exp(z_i/T) /
# sum_j exp(z_j/T), softens the probabilities as T grows -- amplifying the tiny wrong-class
# probabilities (the teacher's "dark knowledge": a 2 looks a bit like a 7, not like a cat) into a
# strong training signal. Higher T -> softer -> more entropy; the top class is unchanged.
import math

def softmax_T(logits, T):
    scaled = [z / T for z in logits]
    m = max(scaled)                                # subtract max for numerical stability
    exps = [math.exp(s - m) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]

def entropy(probs):
    return -sum(p * math.log(p) for p in probs if p > 0)     # softness of the distribution

def kl_divergence(p, q):
    return sum(pi * math.log(pi / qi) for pi, qi in zip(p, q) if pi > 0)   # student vs teacher

def argmax(xs):
    return xs.index(max(xs))

z = [2.0, 1.0, 0.0]
print("softmax T=1 (sharp):", [round(x, 3) for x in softmax_T(z, 1.0)])
print("softmax T=5 (soft): ", [round(x, 3) for x in softmax_T(z, 5.0)])
print("entropy T=1:", round(entropy(softmax_T(z, 1.0)), 3), " T=5:", round(entropy(softmax_T(z, 5.0)), 3))
print("max prob at T=1000 (-> uniform 0.33):", round(max(softmax_T(z, 1000.0)), 3))
print("top class unchanged by T? argmax:", argmax(softmax_T(z, 5.0)))

What it showed

Distillation demonstrated that a model's soft outputs are a far richer teaching signal than labels:

Contribution	Significance
Soft targets beat hard labels	Students trained on the teacher's soft distribution reached accuracy far closer to the teacher than hard-label training.
Temperature exposes dark knowledge	Softening the softmax made the informative wrong-class structure a usable training signal.
Compress ensembles too	An expensive ensemble's combined predictions could be distilled into a single cheap model.
Generalizes better	Distilled students often generalize better than same-size models trained on labels alone.

Distillation isn't magic — the student is still smaller, so there's a ceiling on how much of the teacher it can hold, and results depend on the temperature, the loss balance, and how well the teacher's structure suits the student's capacity. But the central finding is robust and surprisingly general: a trained model's probability outputs encode transferable knowledge that a one-hot label discards, and you can pour it into a smaller vessel.

The compression tool

Knowledge distillation became one of the standard levers for building efficient models. When you need something that runs on a phone, at the edge, or cheaply at scale, distilling a big teacher into a small student is a first-line technique. It's the "Distil" in DistilBERT and the idea behind TinyBERT and a long line of compact, deployable models that retain most of a large model's accuracy at a fraction of the cost — pairing naturally with other efficiency methods like low-rank adaptation and quantization.

The idea also generalized well beyond its original classification setting. Modern small language models are frequently trained on text generated by much larger models — a form of distillation where the teacher's outputs (not its probability vectors but its actual completions) become the student's training data. Microsoft's Phi line leaned on carefully curated and model-generated "textbook-quality" data to make small models punch far above their weight, and "distill a strong teacher into a small student" is now a common recipe across the efficient-LLM landscape. The 2015 insight — that a model's soft outputs carry knowledge the label throws away — turned out to be one of the most quietly influential ideas in making capable AI cheap enough to deploy.

Worth knowing

There's a subtle scaling detail: because the softmax gradients scale roughly as 1/T², the soft-target loss is usually multiplied by T² to keep its gradient magnitude comparable to the hard-label loss when the two are combined. Without that correction, raising the temperature would inadvertently shrink the soft-target term's influence — a small factor that matters for getting distillation to actually work well.

Frequently asked

Quick answers

What is knowledge distillation?

Training a small student model to match a big teacher's soft output distribution, not just the hard label — compressing the teacher's competence into a smaller model.

What is dark knowledge?

The information in the teacher's relative probabilities for the wrong classes — its learned sense of which classes are similar. A one-hot label discards it.

What does temperature do?

Dividing logits by T > 1 before the softmax softens the distribution, amplifying tiny wrong-class probabilities so the dark knowledge becomes a strong signal — without changing the top class.

Why does it matter today?

It's a standard way to build small, fast, cheap models (DistilBERT, TinyBERT) and, generalized, to train small LLMs from a larger model's outputs.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

Knowledge
distillation.

More than a label

Dark knowledge

Turn up the temperature

The softened softmax

Soft targets, from scratch

What it showed

The compression tool

Quick answers

What is knowledge distillation?

What is dark knowledge?

What does temperature do?

Why does it matter today?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Knowledgedistillation.

More than a label

Dark knowledge

Turn up the temperature

The softened softmax

Soft targets, from scratch

What it showed

The compression tool

Quick answers

What is knowledge distillation?

What is dark knowledge?

What does temperature do?

Why does it matter today?

Go deeper

Phi →

BERT →

LoRA →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

Knowledge
distillation.