Beyond English-Centric Multilingual Machine Translation

By Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary and 7 more. 2020

1,098 citations recorded by the source corpus.

Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Recent work (Lewis et al., 2021; Lepikhin et al., 2021; Fedus et al., 2021; Fan et al., 2021) has studied different conditional compute strategies that work well with”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We use the following baseline models where appropriate (e.g. in settings where they support the language of the evaluation dataset): • mGPT (Shliazhko et al., 2022), GPT-style models trained on 60 languages from Wikipedia and Common Crawl • GPT-Neo (Black et al., 2021), GPT-J-6B (Wang and Komatsuzak”
A Review of Sparse Expert Models in Deep Learning (2022, background): “Fan et al. (2021) uses explicit language-speciﬁc sublayers where input tokens can be deterministically routed based on their language.”
FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “In machine translation, the release of new benchmarks like FLoRes-101 [20] has enabled advances in publicly available massively multilingual machine translation systems [21].”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “Perhaps the most common use of sparse layers is in adding language-speciﬁc layers to machine-translation systems (Bapna et al., 2019; Fan et al., 2020), or task-speciﬁc layers to pre-trained language models (Houlsby et al., 2019).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “(ii) M2M-100 is a large multilingual model which supports “many-to-many” translation directions (Fan et al., 2020).”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “The goal of Natural Language Generation (NLG) is to produce fuent text in many domains, such as machine translation [21], summarization [34], dialogue [30], style transfer [18], and programming code [13].”
Designing Effective Sparse Expert Models (2022, positive): “Finally, Fan et al. (2021) designs an architecture with explicit language-speciﬁc sublayers (rather than allowing arbitrary routing as done in Lepikhin et al. (2020)) to yield gains of +1 BLEU.”
Is ChatGPT A Good Translator? A Preliminary Study (2023, background): “• Knowledge Transfer: While parallel data between two distant languages is often scarce (Fan et al., 2021; Wang et al., 2022b), the parallel data between them and the pivot language can be relatively considerable, which is expected to learn better translation ability for source-pivot and pivot-targe”
Tricks for Training Sparse Translation Models (2021, background): “In this work, we study multilingual machine translation as a multi-task learning problem ( Dong et al., 2015; Domhan and Hieber, 2017), where a single model is trained to translate between many language pairs ( Fan et al., 2021).”
A Survey on Mixture of Experts in Large Language Models (2024, positive): “In contrast to explicit language-speciﬁc expert selection, NLLB [76] leverages trainable gating to manage multilingual machine translation tasks, outperforming the M2M-100 approach [83].”