The Missing Ingredient in Zero-Shot Neural Machine Translation

By Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, Wolfgang Macherey. 2019

123 citations recorded by the source corpus.

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “This also indicates why methods that explicitly force languages to share the same representation space (Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018) may be key to improving zero-shot translation performance.”
Complete Multilingual Neural Machine Translation (2020, neutral): “Scaling cMNMT: 12,432 Language Pairs We run additional experiments on a 112 language in-house dataset (Arivazhagan et al., 2019b) to see if our approach scales to 12,432 language pairs.”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018; Tan et al., 2019), improve low-resource translation (Ha et al., 2016; Arivazhagan et al., 2019b), 1We release our code at https://github. com/bzhangGo/zero.”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “In this work, we start with a typical degeneracy issue of zero-shot NMT, reported in several recent works (Arivazhagan et al., 2018; Ses-torain et al., 2018), that zero-shot NMT is sensitive to training conditions, and the translation quality usually lags behind the pivot-based meth-ods which use a…”
Investigating Multilingual NMT Representations at Scale (2019, methodology): “We study multilingual NMT on a massive scale, using an in-house training corpus (Arivazhagan et al., 2019b) generated by crawling and extracting parallel sentences from the web (Uszkoreit et al., 2010).”
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning (2020, methodology): “We follow Arivazhagan et al. (2019b) and use the following temperature based sampling function with temperature T to sample data for each direction: pi,j ∝ ( |Bi,j |∑ i,j |Bi,j | )1/T”
Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “As a result, if the network capacity is large enough, it may partition itself into different sub-spaces for different language pairs (Arivazhagan et al., 2019).”
On the Linguistic Representational Power of Neural Machine Translation Models (2019, background): “in a zero-shot MT scenario (Johnson et al. 2017; Arivazhagan et al. 2019) may also yield new insights for improving such systems.”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, methodology): “To support translation between Indic languages, we also need direct parallel corpus between these languages since zero-shot translation is insufficient to address translation between non-English languages (Arivazhagan et al., 2019a).”
A Survey of Multilingual Neural Machine Translation (2019, background): “Arivazhagan et al. (2018) included explicit language invariance losses in the optimization function to encourage parallel sentences to have the same representation.”
Using natural language prompts for machine translation (2022, unknown): “(Arivazhagan et al., 2019a; Sestorain et al., 2018; Johnson et al., 2017) or zero-resource (Firat et al., 2016) setting.”
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020, methodology): “Future work should explore techniques like iterative back-translation (Hoang et al., 2018) for further improvement and scaling to larger model capacities and more languages (Arivazhagan et al., 2019b; Huang et al., 2019)”
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, background): “Several recent papers focus specifically upon improving the zero-shot performance of multilingual MT models (Chen et al., 2017; Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018; Al-Shedivat and Parikh, 2019; Sestorain et al., 2019).”
Towards Interlingua Neural Machine Translation (2019, result): “Related work (Arivazhagan et al. 2018) shows similar results for the case of a multilingual system with shared encoder and decoder.”
Consistency by Agreement in Zero-Shot Neural Machine Translation (2019, methodology): “Lu et al. (2018) and Arivazhagan et al. (2018) proposed different techniques to obtain “neural in-terlingual” representations that are passed to the decoder.”