Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

By Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, Karthik Raman. 2019

76 citations recorded by the source corpus.

The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model (Aharoni, Johnson, and Firat 2019). Its improved translation performance on low resource languages hints at potential cross-lingual transfer capability for downstream tasks. In this paper, we evaluate the cross-lingual effectiveness of representations from the encoder of a massively multilingual NMT model on 5 downstream classification and sequence labeling tasks covering a diverse set of over 50 languages. We compare against a strong baseline, multilingual BERT (mBERT) (Devlin et al. 2018), in different cross-lingual transfer learning scenarios and show gains in zero-shot transfer in 4 out of these 5 tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “On the other hand, we have also seen that large translation models can be adapted for a variety of downstream tasks (Siddhant et al., 2020), suggesting that specialists can also serve as generalists.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Past work also leverages supervised ASR data to build multilingual representations of speech (Kannan et al., 2019; Bai et al., 2021), similar to how multilingual machine translation in NLP is used to build multilingual representations (Eriguchi et al., 2018; Siddhant et al., 2020).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “, 2019), multilingual extensions of pretrained encoders have recently been shown to be effective for learning deep cross-lingual representations (Eriguchi et al., 2018; Pires et al., 2019; Wu & Dredze, 2019; Lample & Conneau, 2019; Siddhant et al., 2020).”
Pre-training via Paraphrasing (2020, background): “Another strand has trained machine translation systems [McCann et al., 2017, Siddhant et al., 2019], but results in Hu et al. [2020] suggest translation is a less effective pre-training task.”
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, methodology): “Other works using parallel data in this line of work include FIL-TER (Fang et al., 2020), AMBER (Hu et al., 2020a) and, MMTE (Siddhant et al., 2020).”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…(2) Positive cross-lingual transfer: by transferring knowledge from high-resource languages (HRL), multilingual models are able to improve performance on low-resource languages (LRL) on a wide variety of tasks (Pires et al., 2019; Wu & Dredze, 2019; Siddhant et al., 2020; Hu et al., 2020).”
OCNLI: Original Chinese Natural Language Inference (2020, background): “While automatically translated data have proven to be useful in many contexts, such as cross-lingual representation learning (Siddhant et al., 2020), there are well-known issues, especially when used in place of human annotated, quality controlled data.”
A Survey of Multilingual Neural Machine Translation (2019, background): “However, a massively multilingual system also runs into representation bottlenecks [2, 135], where not all translation directions show improved performance despite a massive amount of data being fed to a model with a massive number of parameters.”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, methodology): “On the other hand, some methods directly leverage multilingual parallel corpora (McCann et al., 2017; Eriguchi et al., 2018; Conneau and Lample, 2019; Huang et al., 2019; Siddhant et al., 2020), which gives some degree of supervision implicitly aligning the words in the two languages.”