Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

By Jiatao Gu, Yong Wang, Kyunghyun Cho, Victor O. K. Li. 2019

112 citations recorded by the source corpus.

Zero-shot translation, translating between language pairs on which a Neural Machine Translation (NMT) system has never been trained, is an emergent property when training the system in multilingual settings. However, naive training for zero-shot NMT easily fails, and is sensitive to hyper-parameter setting. The performance typically lags far behind the more conventional pivot-based approach which translates twice using a third language as a pivot. In this work, we address the degeneracy problem due to capturing spurious correlations by quantitatively analyzing the mutual information between language IDs of the source and decoded sentences. Inspired by this analysis, we propose to use two simple but effective approaches: (1) decoder pre-training; (2) back-translation. These methods show significant improvement (4 22 BLEU points) over the vanilla zero-shot translation on three challenging multilingual datasets, and achieve similar or better results than the pivot-based approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “This setup is common for multilingual MT systems (Johnson et al., 2017; Gu et al., 2019).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “This also indicates why methods that explicitly force languages to share the same representation space (Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018) may be key to improving zero-shot translation performance.”
Complete Multilingual Neural Machine Translation (2020, neutral): “Although pivot-based approaches perform suf-ﬁciently good when cascaded with strong bilingual models (Gu et al., 2019), their practicality is limited due to compounding errors from pipelining and doubled inference cost.”
Cross-Lingual Natural Language Generation via Pre-Training (2019, methodology): “Cross-Lingual Auto-Encoding (XAE) If only DAE is used as the pre-training task for the decoder, we found that the model ignores the target language tag while generating just the same language as the input, caused by the spurious correlation issue (Gu et al. 2019).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Two solutions are available to solve this problem: to make the source explicitly target language independent [6, 69, 147, 166], and to improve model capacity [54, 206].”
Multilingual Translation from Denoising Pre-Training (2021, background): “This result echos the findings in (Gu et al., 2019) regarding the importance of decoupling source and target languages encoders and decoders learning in zero-shot translation.”
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning (2020, background): “Training a universal translation system between multiple languages (Firat et al., 2016; Johnson et al., 2017) has shown enormous improvement for translating low-resource languages (Gu et al., 2018), and even enabling zero-shot translation (Gu et al., 2019; Arivazhagan et al., 2019a).”
A Multilingual View of Unsupervised Machine Translation (2020, methodology): “C L ] 2 1 Fe b 20 20 2019; Gu et al., 2019; Al-Shedivat & Parikh, 2019).”
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, background): “Several recent papers focus specifically upon improving the zero-shot performance of multilingual MT models (Chen et al., 2017; Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018; Al-Shedivat and Parikh, 2019; Sestorain et al., 2019).”