Consistency by Agreement in Zero-Shot Neural Machine Translation

By Maruan Al-Shedivat, Ankur P. Parikh. 2019

57 citations recorded by the source corpus.

Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization—a challenging setup that tests models on translation directions they have not been optimized for at training time. To solve the problem, we (i) reformulate multilingual translation as probabilistic inference, (ii) define the notion of zero-shot consistency and show why standard training often results in models unsuitable for zero-shot tasks, and (iii) introduce a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages. We test our multilingual NMT models on multiple public zero-shot translation benchmarks (IWSLT17, UN corpus, Europarl) and show that agreement-based learning often results in 2-3 BLEU zero-shot improvement over strong baselines without any loss in performance on supervised translation directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…more languages (Aharoni et al., 2019), adding regularization to promote cross-lingual transfer (Arivazhagan et al., 2019; Gu et al., 2019; Al-Shedivat and Parikh, 2019), and modeling strategies that encourage shared cross-lingual representations (Lu et al., 2018; Escolano et al., 2019).”
Complete Multilingual Neural Machine Translation (2020, neutral): “Lastly, zero-shot approaches attempt to measure the generalization performance of the MNMT models, but to date, the zero-shot quality still trails behind the pivot and zero-resource meth-ods (Al-Shedivat and Parikh, 2019).”
Investigating Multilingual NMT Representations at Scale (2019, background): “We study multilingual NMT on a massive scale, using an in-house training corpus (Arivazhagan et al., 2019b) generated by crawling and extracting parallel sentences from the web (Uszkoreit et al., 2010).”
Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “In addition, zero-resource techniques which generate artificial data for the missing directions have been proposed as an alternative to zero-shot translation (Chen et al., 2018; Al-Shedivat and Parikh, 2019; Chen et al., 2017).”
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, background): “Several recent papers focus specifically upon improving the zero-shot performance of multilingual MT models (Chen et al., 2017; Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018; Al-Shedivat and Parikh, 2019; Sestorain et al., 2019).”
A Multilingual View of Unsupervised Machine Translation (2020, methodology): “C L ] 2 1 Fe b 20 20 2019; Gu et al., 2019; Al-Shedivat & Parikh, 2019).”