On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

By Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Sta'nczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem. International Conference on Learning Representations, 2023

576 citations recorded by the source corpus.

Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, current KD methods for auto-regressive sequence models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference. To address this issue, we introduce Generalized Knowledge Distillation (GKD). Instead of solely relying on a fixed set of output sequences, GKD trains the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences. Unlike supervised KD approaches, GKD also offers the flexibility to employ alternative loss functions between the student and teacher, which can be useful when the student lacks the expressivity to mimic the teacher's distribution. Furthermore, GKD facilitates the seamless integration of distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for distilling auto-regressive language models on summarization, translation, and arithmetic reasoning tasks, and task-agnostic distillation for instruction-tuning.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a57ef1f5c3af185af79751855b8033b7fc6d89b3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Gemma 3 Technical Report (2025, unknown): “Our post-training approach relies on an improved version of knowledge distillation (Agarwal et al., 2024; Anil et al., 2018; Hin-ton et al., 2015) from a large IT teacher, along with a RL finetuning phase based on improved versions of BOND (Sessa et al., 2024), WARM (Ramé et al., 2024b), and WARP…”
Gemma 2: Improving Open Language Models at a Practical Size (2024, methodology): “We also run distillation from the teacher on the student’s distribution (Agarwal et al., 2024).”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “Similarly, the Feature section discusses how knowledge based on features, such as logits serving as soft labels, can offer deeper, intrinsic insights into the teacher model (Gu et al., 2024; Agarwal et al., 2024).”
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026, unknown): “For example, GKD [20] reduces exposure bias with on-policy sampling, while MiniLLM [40] and DistiLLM [41] improve distribution matching through stabilized KL objectives.”