Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

By Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Z. Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Gregory S. Corrado and 2 more. 2016

2,235 citations recorded by the source corpus.

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT systems using a single model. On the WMT’14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-theart results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT’14 and WMT’15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. Our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and also show some interesting examples when mixing languages.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “Dataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs.”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “For MT, the most relevant field is multilingual translation (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Arivazhagan et al., 2019) where the ultimate goal is to jointly train one translation model that translates multiple language directions at the same time, and shares…”
Massively Multilingual Neural Machine Translation (2019, methodology): “While the mentioned studies provide valuable contributions to improving multilingual models, they apply their models on only up to 7 languages (Johnson et al., 2017) and 20 trained directions (Cettolo et al., 2017) in a single model, whereas we focus on scaling NMT to much larger numbers of…”
Cross-lingual Language Model Pretraining (2019, methodology): “But the most successful recent approach of cross-lingual encoders is probably the one of Johnson et al. (2017) for multilingual machine translation.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “The proposed MetaNMT differs from the existing framework of multilingual translation (Lee et al., 2016; Johnson et al., 2016; Gu et al., 2018b) or transfer learning (Zoph et al., 2016).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Multitask learning over different machine translation language pairs can enable zero-shot translation (Johnson et al., 2017), and sequence-to-sequence architectures can be used to multitask across translation, parsing, and image captioning (Luong et al.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “…Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al., 2015; Firat et al., 2016; Johnson et al., 2016; Zoph et al., 2016; Neubig and Hu, 2018; Aharoni et al., 2019; Ari-vazhagan et al., 2019).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “This is similar to how codes and natural language sequences have been used in multi-task settings (Wu et al., 2016; Johnson et al., 2017; McCann et al., 2018) to control conditional language models.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Unfortunately, this simultaneously results in performance degradation on high-resource languages due to interference and constrained capacity (Johnson et al., 2017; Tan et al., 2019).”
Parameter-Efficient Transfer Learning for NLP (2019, background): “, 2017), machine translation (Johnson et al., 2017), and question answering (Choi et al.”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “Examples include the language-indicators, <2ja> and <2de> , for multilingual U+41 U+302 U+300 <tab> U+1EA6 U+41 U+302 U+301 <tab> U+1EA4 ... Figure 2: Custom normalization rule in TSV models (Johnson et al., 2016).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “While notable recent approaches have considered building a shared sentence encoder for multiple languages using publicly available parallel corpora (Johnson et al., 2016; Schwenk et al., 2017; España-Bonet et al., 2017), the lack of a large-scale, sentence-level semantic evaluation has limited their”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “In NLP, zero-shot learning work also includes translating between unseen language pairs (Johnson et al., 2017; Pham et al., 2019), language modeling on unseen languages (Lauscher et al.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “However, in the past, these systems have not performed as well as bilingual models when trained on the same language pairs (Johnson et al., 2017), as model capacity necessarily ∗.”
Complete Multilingual Neural Machine Translation (2020, neutral): “Multilingual machine translation (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019), which can serve multiple language pairs with a single model, has attracted much attention.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “Language Indicator Inspired by (Johnson et al., 2017; Ha et al., 2016 Figure 1: The proposed mRASP method.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “We chose multilingual neural machine translation (MT) [39, 40, 41] to validate our design for efficient training with GShard.”
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, background): “This follows precedent set in Johnson et al. (2017) and Zhou et al. (2019) for controlling translations via tag-based annotations.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, methodology): “Additionally, recent progress in NMT has enabled one to train multilingual NMT systems that support translation from multiple source languages into multiple target languages within a single model (Johnson et al. 2017).”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “Recent research in NMT for agglu-tinative languages found that morphological segmentation outperforms the most widely used segmentation technique, byte-pair encoding (BPE, using character sequence frequencies) (Sennrich et al., 2016).”