Autoregressive Knowledge Distillation through Imitation Learning
By Alexander Lin, Jeremy Wohlwend, Howard Chen, Tao Lei. 2020
72 citations recorded by the source corpus.
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and summarization, our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nino Vieillard
- Matthieu Geist
- Olivier Bachem
- Piotr Sta'nczyk
- Yongchao Zhou
- Rishabh Agarwal
- Sabela Ramos
- Gabriel Recchia
In Computer Vision and Pattern Recognition
- Nino Vieillard
- Matthieu Geist
- Olivier Bachem
- Gabriel Recchia
- Bernhard Schölkopf
- Francesco Locatello
- Stefan Bauer
- Michael Tschannen
In Natural Language Processing
- Nino Vieillard
- Matthieu Geist
- Olivier Bachem
- Piotr Sta'nczyk
- Yongchao Zhou
- Rishabh Agarwal
- Sabela Ramos
- Gabriel Recchia
Citations in the corpus
- Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “Finally, the use of ‘environment forcing’ (Section 4.2) is inspired by techniques for minimizing exposure bias [33]–[35] and corresponds to the conditioning on sequences of prior actions and states undertaken by [17], [18] when using ‘decision transformers’ and ‘trajectory transformers’ in…”
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, methodology): “Notably, on-policy GKD on the 5% subsampled dataset, without any ground-truth summaries, outperforms supervised KD and ImitKD with entire training dataset with ground-truth summaries.”