Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

By Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, M. Chen, Yuan Cao, George F. Foster, Colin Cherry and 3 more. 2019

477 citations recorded by the source corpus.

We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair. We set a milestone towards this goal by building a single massively multilingual NMT model handling 103 languages trained on over 25 billion examples. Our system demonstrates effective transfer learning ability, significantly improving translation quality of low-resource languages, while keeping high-resource language translation quality on-par with competitive bilingual baselines. We provide in-depth analysis of various aspects of model building that are crucial to achieving quality and practicality in universal NMT. While we prototype a high-quality universal translation system, our extensive empirical analysis exposes issues that need to be further addressed, and we suggest directions for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “To stress test our adapters based approach, we apply this to a massively multilingual translation task on a real world dataset (Arivazhagan et al., 2019).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “The diverse capabilities of language models at scale have drawn attention to the tradeoffs between specialist models (one model per task) and generalist models (one model for many tasks; Arivazhagan et al., 2019; Pratap et al., 2020), for which our study has potential implications.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “…objective to 10 since we observe fewer characters per token than T5, (iv) adjust the language sampling exponent α to {0.2, 0.7} as used in MMNMT (Arivazhagan et al., 2019) and mBERT (Devlin, 2018), respectively, (v) turn off the “line length filter” in the mC4 data pipeline, and (vi)…”
Beyond English-Centric Multilingual Machine Translation (2020, background): “We describe Sinkhorn Temperature Sampling, which extends the temperature sampling strategy to the Many-to-Many setting.”
Complete Multilingual Neural Machine Translation (2020, positive): “Scaling cMNMT: 12,432 Language Pairs We run additional experiments on a 112 language in-house dataset (Arivazhagan et al., 2019b) to see if our approach scales to 12,432 language pairs.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “, deeper ResNets performed better [8], bigger Transformer models achieved better translation quality [10], models with larger vocabulary, or embedding or feature crosses work better, too [14, 13].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “We upsample lower resource languages using temperature-based sampling (Arivazhagan et al., 2019), with T = 3.0.”
Scaling Laws for Neural Machine Translation (2021, background): “Neural machine translation (NMT) has long enjoyed the benefits of scaling [19, 4, 25], but studies investigating the scaling behavior of NMT models are missing.”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “For the simultaneous ST setting, it includes online models with widely used policies: monotonic attention (Raffel et al., 2017), wait-k (Ma et al., 2019), monotonic inﬁnite lookback attention (Arivazhagan et al., 2019b), and mono-tonic multihead attention (Ma et al., 2020b).”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “The work most similar to ours is Arivazhagan et al. (2019), which trains a single model in 103 languages on over 25 billion parallel sentences.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…language processing (NLP), there is a pressing urgency to build systems that serve all of the world’s approximately 6,900 languages to overcome language barriers and enable universal information access for the world’s citizens (Ruder et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019b).”
Nearest Neighbor Machine Translation (2020, methodology): “Finally, various studies have explored retrieving additional information to improve domain adaptation, often using lexicons (Hu et al., 2019), domain-adaptive training (Farajian et al., 2017) or attending over neighbors similar to n -grams in the source (Bapna & Firat, 2019).”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “They also play a role in multi-task learning, re-inforcement learning and standard supervised learning, for example under distribution shift (Arivazhagan et al., 2019; Toneva et al., 2018).”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, methodology): “Within each source, we use a temperature-based sampling scheme based on the numbers of samples of the relevant datasets with a temperature of 5 (Arivazhagan et al., 2019).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Such systems are beneficial not only because they can handle multiple translation directions using a single model, and thus reduce training and maintenance costs, but also due to joint training with high-resource languages they can improve performance on low- and zero-resource languages (Arivazhagan”