Ensemble Distillation for Neural Machine Translation

By Markus Freitag, Yaser Al-Onaizan, Baskaran Sankaran. 2017

114 citations recorded by the source corpus.

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tie-Yan Liu
2. Tao Qin
3. Xu Tan
4. Zhou Zhao
5. Yi Ren
6. Dacheng Tao
7. Sheng Zhao
8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Tao Qin
3. Tie-Yan Liu
4. Luc Van Gool
5. Kyunghyun Cho
6. Xiaogang Wang
7. Trevor Darrell
8. Xu Tan

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Kyunghyun Cho
4. Xu Tan
5. Felix Stahlberg
6. Graham Neubig
7. Di He
8. Raj Dabre

Citations in the corpus

• The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Iterative Knowledge Distillation and 8 models combination yielded another +0.8/1.4 BLEU points.”
• Knowledge Distillation: A Survey (2020, methodology): “To improve the translation quality, the knowledge distillation method was proposed by adopting an ensemble teacher model with a data filtering method to teach the student model, and the ensemble teacher model was based on all kinds of NMT models (Freitag et al., 2017).”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “, 2015) and natural language processing (Kim & Rush, 2016a; Freitag et al., 2017).”
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation (2020, background): “Freitag, Al-Onaizan, and Sankaran (2017) took a step further and expanded it to a multi-task scenario.”
• Balancing Cost and Benefit with Tied-Multi Transformers (2020, methodology): “In this paper, we counter such drops with knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016; Freitag et al., 2017).”
• GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, methodology): “We follow the data filtering and pre-processing methods described in [16, 17], and utilize iterative sequence-level knowledge distillation [18, 19] and back translation [20] techniques to improve the performance of MT.”
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding (2018, methodology): “a pseudo-parallel corpus generated from the underlying NMT model (Gao and He, 2013; Auli and Gao, 2014; Kim and Rush, 2016; Chen et al., 2017; Freitag et al., 2017; Zhang et al., 2017) for actor training.”
• Pieces of Eight: 8-bit Neural Machine Translation (2018, methodology): “The English-German model was run with both a single model (1x) and an ensemble of two models (2x) (Freitag et al., 2017).”
• Data Diversification: A Simple Strategy For Neural Machine Translation (2019, methodology): “In this context, our method also differs from the ensemble knowledge distillation method [7], which uses the teachers to jointly generate a single version of data.”
• Neural Machine Translation: A Review (2019, methodology): “• NMT model shrinking and reduced precision (Wu et al., 2016; See et al., 2016; Kim & Rush, 2016; Freitag et al., 2017; Zhang et al., 2018)”
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “Nor can they explain why the superior performance of ensemble can also be distilled into a single model using knowledge distillation [21, 23, 30, 35, 43, 61, 91].”
• Knowledge Distillation from Bert in Pre-Training and Fine-Tuning for Polyphone Disambiguation (2019, background): “lot of works expand the usage of knowledge distillation to a variety of tasks, such as image classification [23, 24, 25] and natural language processing [26, 27, 28].”