Distillation transfers ability from a big model to a small one by training the student on the teacher’s outputs (or output distributions). It is how production systems get frontier-adjacent quality at small-model prices — and a big reason capable small models keep appearing. Contrast with quantization, which shrinks the same model’s numbers rather than training a new one.
Worked example: a small student trains on a large teacher’s output probabilities — the ‘soft’ targets (say 70% cat, 20% dog) carry more signal than a one-hot label. Gotcha: the student can approach but not exceed the teacher on the distilled behaviour, and it inherits the teacher’s blind spots — distillation copies capability, it does not create new knowledge.