Understanding Knowledge Distillation in Non-autoregressive Machine Translation
By Chunting Zhou, Graham Neubig, Jiatao Gu. 2019
237 citations recorded by the source corpus.
Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.
Subfields: Artificial Intelligence, Information Systems, Efficiency & Compression, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Noah A. Smith
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
In Information Systems
In Efficiency & Compression
- Kurt Keutzer
- A. Gholami
- Michael W. Mahoney
- Sehoon Kim
- Dacheng Tao
- Chitwan Saharia
- Jiatao Gu
- William Chan
In Machine Translation
Citations in the corpus
- Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, background): “This is because KD reduces the complexity of training targets (by replacing the gold training target with a teacher model generated target), which helps with low-capacity adapter (with 4% parameter) by providing an easier translation/summarization task (Zhou et al., 2019).”
- Scaling Laws for Neural Machine Translation (2021, background): “We believe the same phenomenon is also related to the "data simplification" effect seeked by non-autoregressive models in NMT (Zhou et al., 2021).”
- Knowledge Distillation: A Survey (2020, methodology): “…2019; Yang et al. 2020c; Tang et al. 2019; Hu et al. 2018; Sun et al. 2019; Nakashole and Flauger 2017; Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al. 2019; Turc et al. 2019; Arora et al. 2019; Clark et al. 2019; Kim and Rush 2016; Mou et al. 2016; Liu et al. 2019f;…”
- Lifelong Language Knowledge Distillation (2020, methodology): “…output sequence ˆ x from the teacher model as the hard target, just like normal language modeling: Seq-KD is usually applied for improving weak non-autoregressive translation (NAT) models (Zhou et al., 2020) by reducing the multi-modality problem in machine translation datasets (Gu et al., 2018).”
- Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, background): “, 2015) are two commonly recognized techniques in training deep neural networks and have been applied in many state-of-the-art models, such as language translation (Vaswani et al., 2017; Tan et al., 2019; Zhou et al., 2020), image classification (Xie et al.”
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “As analyzed in Zhou et al. (2020), KD is able to simplify the training data where the generated targets have less noise and are aligned to the inputs more de-terministically.”
- Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “We opted for self-training through forward translation in this work, following prior work that showed that unfiltered translation can be effective in NMT for model compression (Kim and Rush, 2016), non-autoregressive translation (Zhou et al., 2020), and domain adaptation (Currey et al., 2020).”
- Large Language Models Are Reasoning Teachers (2022, methodology): “Related KD approaches have also been used to improve the performance of non-autoregressive translation by training on output generated by an autoregressive translation model (Zhou et al., 2020).”
- Rule-based Morphological Inflection Improves Neural Terminology Translation (2021, methodology): “We evaluate translation quality using sacreBLEU (Post, 2018).23 22As shown in prior work, the batch sizes for training non-autoregressive models are typically larger than the AR model (Zhou et al., 2020).”
- Deep Streaming Label Learning (2020, methodology): “, 2018), machine translation (Zhou et al., 2020), and meta-learning (Bai et al.”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Recent studies [225], [226] show that knowledge distillation might become less effective when there exists a large capacity gap between the teacher and student models.”
- Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, background): “This likely arises because model translations are simpler than human ones, which aids learning (Zhou et al., 2020).”
- ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020, background): “Another related approach trains an “actor” network to manipulate the hidden state of an autoregressive neural MT system (Gu et al., 2017; Chen et al., 2018; Zhou et al., 2020) in order to bias it toward outputs with better BLEU scores.”
- Autoregressive Knowledge Distillation through Imitation Learning (2020, unknown): “have used seqKD for non-autoregressive translation models (Gu et al., 2017; Zhou et al., 2019), lowresource settings (Chen et al.”
- Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation (2020, methodology): “…families; we depart from previous practice where NAR models trained with distillation are compared with AR models trained without (Ran et al., 2021; Sun et al., 2019; Shu et al., 2020; Zhou et al., 2020; Saharia et al., 2020) with a few exceptions (Ghazvinine-jad et al., 2019; Kasai et al., 2020).”
- Big Little Transformer Decoder (2023, background): “In order to achieve comparable performance to that of autoregressive processes, nonautoregressive decoding generally requires complex and often taskdependent training strategies, supplementary hinting information that serves as a guide for the decoding process [35, 49, 55, 65, 66], and knowledge dis”
- Non-Autoregressive Machine Translation with Latent Alignments (2020, result): “This is consistent to prior work finding distillation to improve model quality (Gu et al., 2018; Zhou et al., 2020).”