Distillation
also: knowledge distillation · teacher-student
Training a small “student” model to imitate a large “teacher” — capability at a fraction of the cost.
Distillation transfers ability from a big model to a small one by training the student on the teacher’s outputs (or output distributions). It is how production systems get frontier-adjacent quality at small-model prices — and a big reason capable small models keep appearing. Contrast with quantization, which shrinks the same model’s numbers rather than training a new one.