Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

By Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich. 2020

456 citations recorded by the source corpus.

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Complete Multilingual Neural Machine Translation (2020, positive): “We ﬁnd the study by (Zhang et al., 2020) closest to our work, having the goal of any-to-any multilingual translation.”
Nearest Neighbor Machine Translation (2020, neutral): “Datastore size and the increase in BLEU are only weakly correlated across languages, though within a language, a larger datastore is decidedly (2)Evaluating our model on the recently released OPUS100 (Zhang et al., 2020) corpus improves upon the result in Zhang et al.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “However, with multilingual parallel data being also Anglo-centric, the capabilities to translate from English versus into English vastly diverge (Zhang et al., 2020).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, methodology): “Note that OPUS-100 only covers 80 languages in F LORES -101, so this ﬁgure is on the subset of 80 languages covered by all models, for comparability.”
End-to-end Speech Translation via Cross-modal Progressive Training (2021, methodology): “MT datasets We use external WMT machine translation datasets(2) [28] for En-De/Es/Fr/Ro/Ru directions, and OPUS100(3) [29] for En-It/Nl/Pt directions.”
DeepNet: Scaling Transformers to 1,000 Layers (2022, background): “With Lemma 1, the query and key projection do not change the bound of expected magnitude of attention update.”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “Training data in WMT-14 is more imbalanced across different language pairs compared to OPUS-100.”
Multilingual Translation from Denoising Pre-Training (2021, methodology): “Compared to other datasets such as OPUS100, we choose to use high quality existing evaluation datasets rather than use part of the training data as evaluation.”
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (2020, methodology): “Further, all query and prediction translations in our approaches use Zhang et al. (2020)’s open source many-to-many, encoder-decoder machine translation system, trained on the OPUS multilingual corpus, covering 100 languages.”
Deep Transformers with Latent Depth (2020, methodology): “The Transformer model has achieved the state-of-the-art performance on various natural language preprocessing (NLP) tasks, originally in neural machine translation [30], and recently in massive multilingual machine translation [3, 37], crosslingual pretraining [8, 17], and many other tasks.”
TorchScale: Transformers at Scale (2022, methodology): “The experiments are conducted using a combination of CCMatrix [Schwenk et al., 2021], CCAligned [El-Kishky et al., 2020], OPUS [Zhang et al., 2020], and Tatoeba 2 .”
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, methodology): “OPUS-100 is a dataset of 55M translations covering 100 languages (198 language pairs, either into or from English).”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Over the last few years, there has been an explosion of work focusing on massively multilingual models that could translate between around 100 languages (Neubig & Hu, 2018; Aharoni et al., 2019b; Arivazhagan et al., 2019; Zhang et al., 2020; Tang et al., 2021; Fan et al., 2021).”
Using natural language prompts for machine translation (2022, unknown): “6 We gather this data from WMT, the OPUS 100 (Zhang et al., 2020) dataset, Paracrawl7 and Samanantar (Ramesh et al.”
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning (2020, background): “Previous work (Aharoni et al., 2019; Arivazhagan et al., 2019b; Zhang et al., 2020) has explored multilingual translation by training multiple directions within the same model from scratch, but this approach faces challenges for mid to low resource languages.”