On-Policy Distillation
Distilling a teacher into a student using the student’s own sampled outputs, rather than a fixed teacher-generated dataset.
On-policy distillation trains a smaller student to imitate a teacher, but on the student’s own generated trajectories: the student samples, the teacher grades or corrects those samples, and the student learns from them. Matching the distribution the student actually produces avoids the train/deploy mismatch of copying a fixed off-policy dataset.
Worked example: distilling a student from a teacher using the STUDENT’s own generated outputs (on-policy rollouts) scored/corrected by the teacher, rather than a fixed dataset of teacher outputs — so the student learns on the exact distribution of mistakes it actually makes. Gotcha: off-policy distillation (train on teacher transcripts) is cheaper but suffers exposure bias — the student never practices recovering from its own errors — while on-policy is costlier (generate then grade each step) but fixes the states the student really visits, the usual quality/cost tradeoff.