Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

By Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, M. Chen, Sneha Kudugunta, Naveen Arivazhagan, Yonghui Wu. 2020

88 citations recorded by the source corpus.

Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged. The first focuses on utilizing high-resource languages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self-supervision to pre-train translation models, followed by fine-tuning on small amounts of supervised data. In this work, we join these two lines of research and demonstrate the efficacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data significantly boosts the translation quality of low-resource languages in multilingual models. (ii) Self-supervision improves zero-shot translation quality in multilingual models. (iii) Leveraging monolingual data with self-supervision provides a viable path towards adding new languages to multilingual models, getting up to 33 BLEU on ro-en translation without any parallel data or back-translation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “Using backtranslation thus requires the ability to translate in both directions, which fits well into the setting of multilingual machine translation (Zhang et al., 2020; Siddhant et al., 2020).”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “The data was processed in the same way as outlined in Siddhant et al. (2020).”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, methodology): “We train our models leveraging both monolingual and parallel datasets, following previous work (Siddhant et al., 2020; Garcia et al., 2020).”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, neutral): “…Pfeiffer et al., 2020): for example, some approaches appear capable of extending machine translation models to new languages with only monolingual data (Artetxe et al., 2017; Lample et al., 2017; Siddhant et al., 2020), and similar ﬁndings have been reported for other NLP tasks (Hu et al., 2020).”
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning (2020, background): “A closely related concurrent work, Siddhant et al. (2020) shows it is possible to train a multilingual system jointly with monolingual datasets based on Song et al. (2019).”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, unknown): “We collect these datasets following Siddhant et al. (2020).”
Multilingual Translation from Denoising Pre-Training (2021, background): “A closely related concurrent work, Siddhant et al. (2020) shows it is possible to train a multilingual system jointly with monolingual datasets based on Song et al. (2019).”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, methodology): “For our experiments, we use parallel training and evaluation data from the WMT corpus and adopt the setup used by (Siddhant et al., 2020) with 15 languages, to and from English.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, background): “…2018; Devlin et al., 2019), multilingual extensions of pre-trained encoders have been proven effective in learning deep contextual cross-lingual representations (Eriguchi et al., 2017; Lample and Conneau, 2019; Wu and Dredze, 2019b; Siddhant et al., 2020; Pires et al., 2019; Pfeiffer et al., 2020).”
Counter-Interference Adapter for Multilingual Machine Translation (2021, methodology): “Johnson et al. (2017) ﬁrst proposed training a single multilingual model via an additional target language tag, which became the paradigm for training multilingual MT models henceforth (Gu et al., 2018; Tan et al., 2018, 2019; Siddhant et al., 2020).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “[167] used the MASS objective [171] for this purpose, Wang et al.”