Multilingual Neural Machine Translation with Knowledge Distillation

By Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, Tie-Yan Liu. 2019

267 citations recorded by the source corpus.

Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “, 2017) is a challenging task in artificial intelligence, and covers a variety of language generation applications such as NMT (Cho et al., 2014; Bahdanau et al., 2015a; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Tan et al., 2019; Artetxe et al., 2017; Lample et al., 2017; 2018; He”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, background): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without signiﬁcant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Unfortunately, this simultaneously results in performance degradation on high-resource languages due to interference and constrained capacity (Johnson et al., 2017; Tan et al., 2019).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “In NLP, Tan et al. (2019) distill single-language-pair machine translation systems into a many-language system.”
Microsoft Research Asia’s Systems for WMT19 (2019, neutral): “, 2016) and knowledge distillation (Kim and Rush, 2016; Tan et al., 2019).”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “They uncover the high and low resource trade-off and the problem of capacity dilution (Johnson et al., 2017; Tan et al., 2019).”
Patient Knowledge Distillation for BERT Model Compression (2019, neutral): “Tan et al. (2019) tried knowledge distillation for multilingual translation.”
Knowledge Distillation: A Survey (2020, methodology): “To obtain lightweight NMT, there are many extended knowledge distillation methods for neural machine translation (Hahn and Choi 2019; Zhou et al. 2019a; Kim and Rush 2016; Gordon and Duh 2019; Wei et al. 2019; Freitag et al. 2017; Tan et al. 2019).”
A Survey on Neural Speech Synthesis (2021, background): “[204] conduct teacher-student distillation [108, 160, 337] to reduce the exposure bias problem, where the teacher is trained with teacher-forcing mode, and the student takes the previously predicted value as input and is optimized to reduce the distance of hidden states between the teacher and stude”
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, methodology): “The total loss of our method is the weighted combination of the original negative log-likelihood loss and the knowledge distillation loss [6, 8] on the labeled data, as well as the knowledge distillation loss on the unlabeled data: where each loss term is formulated in Equation 1, 4 and 6, λ is the…”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “Tan et al. [35] exploited KD for multi-lingual machine translation.”
Multilingual Neural Machine Translation with Language Clustering (2019, background): “Multilingual NMT NMT has recently been extended from the translation of a language pair to multilingual translation (Dong et al., 2015; Luong et al., 2015a; Firat et al., 2016; Lu et al., 2018; Johnson et al., 2017; He et al., 2019; Ha et al., 2016; Platanios et al., 2018; Tan et al., 2019).”
Knowledge Inheritance for Pre-trained Language Models (2021, unknown): “We show that using the information of top-K logits (Tan et al., 2019) can reduce the memory footprint without much performance decrease.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Multilingual models learn joint representations across languages to the extent of the parameters being shared (Wu and Dredze, 2019; Tiedemann, 2018; Tan et al., 2019; Zhang et al., 2020; Östling and Tiedemann, 2016; Kudugunta et al., 2019).”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, methodology): “Distillation and Data Augmentation Our approach of using data generated from code-davinci-002 to tune smaller FlanT5 can be viewed as either distillation (Tan et al., 2019) or data augmentation (Li et al.”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “[142] trained bilingual models for all language pairs involved and then these bilingual models are used as teacher models to train a single student model for all language pairs.”
Knowledge Distillation from Bert in Pre-Training and Fine-Tuning for Polyphone Disambiguation (2019, background): “lot of works expand the usage of knowledge distillation to a variety of tasks, such as image classification [23, 24, 25] and natural language processing [26, 27, 28].”
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, methodology): “Therefore, we further leverage knowledge distillation [18, 37], which generates target sequences given source sequences as input to construct a pseudo corpus, to customize the TTS and ASR models for better accuracy.”