Massively Multilingual Neural Machine Translation

By Roee Aharoni, Melvin Johnson, Orhan Firat. 2019

549 citations recorded by the source corpus.

Multilingual Neural Machine Translation enables training a single model that supports translation from multiple source languages into multiple target languages. We perform extensive experiments in training massively multilingual NMT models, involving up to 103 distinct languages and 204 translation directions simultaneously. We explore different setups for training such models and analyze the trade-offs between translation quality and various modeling decisions. We report results on the publicly available TED talks multilingual corpus where we show that massively multilingual many-to-many models are effective in low resource settings, outperforming the previous state-of-the-art while supporting up to 59 languages in 116 translation directions in a single model. Our experiments on a large-scale dataset with 103 languages, 204 trained directions and up to one million examples per direction also show promising results, surpassing strong bilingual baselines and encouraging future work on massively multilingual NMT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Models are Few-Shot Learners (2020, methodology): “l computation framework [BLC13] and speciﬁcally, the mixture-of-experts method [SMM+17] has been used to produce 100 billion parameter models and more recently 50 billion parameter translation models [AJF19], though only a small fraction of the parameters are actually used on each forward pass. A th”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “For MT, the most relevant field is multilingual translation (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Arivazhagan et al., 2019) where the ultimate goal is to jointly train one translation model that translates multiple language directions at the same time, and shares…”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “…Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al., 2015; Firat et al., 2016; Johnson et al., 2016; Zoph et al., 2016; Neubig and Hu, 2018; Aharoni et al., 2019; Ari-vazhagan et al., 2019).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “We also notice that the zero-shot performance for most language pairs increases as we move from the 10 language model to the 102 language model , possibly due to the regularization effect in the capacity constrained setting, similar to what was observed in (Aharoni et al., 2019).”
Learning Compact Metrics for MT (2021, neutral): “The phenomenon has been well-studied for tasks such as translation (Aharoni et al., 2019) and language inference (Conneau et al., 2020b), but it is less well understood in the context of evaluation metrics.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “This has been alleviated by increasing model capacity (Aharoni et al., 2019; Zhang et al., 2020), but increased model size also necessitates larger multilingual training datasets which are laborious and difficult to create.”
Complete Multilingual Neural Machine Translation (2020, neutral): “Multilingual machine translation (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019), which can serve multiple language pairs with a single model, has attracted much attention.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “We chose multilingual neural machine translation (MT) (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019) to validate our design for efﬁcient training with GShard.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Recent progress in NMT has enabled one to train multilingual systems that support translation from multiple source languages into multiple target languages within a single model (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019).”
Energy and Policy Considerations for Modern Deep Learning Research (2020, background): “…many application areas of artificial intelligence, such as game playing (Silver et al. 2017; OpenAI 2018), natural language processing (Devlin et al. 2019; Aharoni, Johnson, and Firat 2019), computer vision (Chollet 2017; Brock, Donahue, and Simonyan 2019) and robotics (Agostinelli et al. 2019).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…language processing (NLP), there is a pressing urgency to build systems that serve all of the world’s approximately 6,900 languages to overcome language barriers and enable universal information access for the world’s citizens (Ruder et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019b).”
The State and Fate of Linguistic Diversity and Inclusion in the NLP World (2020, background): “…the Xs and Ys; yet, some of the most recent techniques on zero-shot learning of massively multilingual systems (Devlin et al., 2019; Conneau and Lample, 2019; Aharoni et al., 2019; Artetxe and Schwenk, 2019) bridge this gap by obliterating the need for large labeled datasets in all languages.”
Masked Language Model Scoring (2019, background): “We consider 5 low-resource directions from the TED Talks dataset (Qi et al., 2018): Arabic (ar), Galician (gl), and Slovak (sk) to English; and English to Arabic, German (de), languages which were considered in Aharoni et al. (2019).”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “The longstanding goal of multilingual machine translation (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Gu et al., 2018) has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, background): “In particular, Aharoni et al. (2019) identi-ﬁed 4 very low-resource pairs ( < 70k); we took the two (gl → en, sk → en) that were not extremely low ( ≤ 6k).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “(Aharoni et al., 2019; Arivazhagan et al., 2019) train models to translate between English and 102 languages, for the 10 most high-resourced African languages on private data, and otherwise on public TED talks (Qi et al., 2018).”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “98 BLEUzero, 1 ©) under the massively multilingual setting (Aharoni et al., 2019), although translations into different target languages show varied performance.”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, neutral): “This is consistent with results in multilingual machine translation (MT) and automatic speech recognition (ASR) where adding more languages can also lead to degradation for highresource languages (Aharoni et al., 2019; Pratap et al., 2020).”