Sequence-Level Knowledge Distillation

By Yoon Kim, Alexander M. Rush. 2016

1,403 citations recorded by the source corpus.

Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches. However to reach competitive performance, NMT models need to be exceedingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT. We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search. Applying weight pruning on top of knowledge distillation results in a student model that has 13 times fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “This idea has previously been applied to sequence-level knowledge distillation by Kim and Rush (2016) and makes the training procedure similar to back-translation (Sennrich et al., 2016).”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Pre-training language models then ﬁne-tuning on downstream tasks has become a new paradigm for natural language processing (NLP).”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “We also follow prior work (Gu et al., 2018; Stern et al., 2018, 2019; Lee et al., 2018) in using distillation (Hinton et al., 2015; Kim and Rush, 2016) to train our models.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Distilling large models into small models (Kim and Rush, 2016; Mou et al., 2016) or ensembles of models into single models (Kuncoro et al., 2016; Liu et al., 2019a) has been shown to improve re-sults for many NLP tasks.”
Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “In this step we generate more pseudo bi-text by sequence level knowledge distillation (Kim and Rush, 2016) apart from using back translation.”
Tagged Back-Translation (2019, methodology): “Beyond the scope of this work, we brieﬂy mention alternative techniques leveraging monolingual data, like forward translation (Uefﬁng et al., 2007; Kim and Rush, 2016), or source copying (Currey et al., 2017).”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Our final submissions consisted of four Deep Transformer models strengthened by Knowledge Distillation, including DLCL25, DLCL30, DLCL25RPR and DLCL30RPR for EN→RU.”
Non-Autoregressive Neural Machine Translation (2017, methodology): “Thus we additionally apply sequence-level knowledge distillation (Kim & Rush, 2016) to construct a new corpus by training an autoregressive machine translation model, known as the teacher, on an existing training corpus, then using that model’s greedy outputs as the targets for training the…”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “In knowledge distillation, the student model often deteriorates in performance (Hinton et al., 2015; Kim and Rush, 2016) compared to its teacher.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “In this paper, we propose to use residual adapter layers (Houlsby et al., 2019) and task embeddings for modelling the aforementioned task-speciﬁc parameters, and we explore different training strategies such as distillation (Hinton et al., 2015; Kim and Rush, 2016).”
Scaling Laws for Neural Machine Translation (2021, neutral): “Moreover, many popular data augmentation algorithms such as Back Translation (Sennrich et al., 2016), sequence level distillation (Kim & Rush, 2016) and self training (He et al., 2020) purposefully add machine generated data into the training pipeline in order to take advantage of monolingual data.”
Accelerating Large Language Model Decoding with Speculative Sampling (2023, methodology): “• Using sequence level distillation (Kim and Rush, 2016) to generate a second model which predicts 𝐾 tokens in parallel.”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, neutral): “The practice of knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016), in which one trains a model on the outputs of another model, has been shown to improve performance on a variety of tasks, potentially even when teacher and student models have the same architecture and model size…”
Are Sixteen Heads Really Better than One? (2019, methodology): “More recently, ﬁne-grained pruning approaches have been popularized by See et al. (2016) and Kim and Rush (2016) (mostly on NMT).”
Born Again Neural Networks (2018, methodology): “, 2014) and a smaller model from (Kim et al., 2016) combining a convolutional layers, highway layers, and a 2-layer LSTM (referred to as CNN-LSTM).”
Masked Language Model Scoring (2019, background): “Future work could explore ﬁnetun-ing to positionwise distributions, as in word-level knowledge distillation (Kim and Rush, 2016), for which our results are a na¨ıve performance bound.”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, methodology): “In the NLP literature, it has previously been used in neural machine translation (Kim and Rush, 2016) and language modeling (Yu et al., 2018).”
Automatically Learning Data Augmentation Policies for Dialogue Tasks (2019, neutral): “…2017; Lemley et al., 2017) and NLP (F¨urstenau and Lapata, 2009; Sen-nrich et al., 2016; Wang and Yang, 2015; Zhang et al., 2015; Jia and Liang, 2016; Kim and Rush, 2016; Hu et al., 2017; Xu et al., 2017; Xia et al., 2017; Silfverberg et al., 2017; Kaﬂe et al., 2017; Hou et al., 2018; Wang et al.,…”