Zero-Resource Translation with Multi-Lingual Neural Machine Translation

By Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, Kyunghyun Cho. 2016

285 citations recorded by the source corpus.

In this paper, we propose a novel finetuning algorithm for the recently introduced multi-way, mulitlingual neural machine translate that enables zero-resource machine translation. When used together with novel many-to-one translation strategies, we empirically show that this finetuning algorithm allows the multi-way, multilingual model to translate a zero-resource language pair (1) as well as a single-pair neural translation model trained with up to 1M direct parallel sentences of the same language pair and (2) better than pivot-based translation strategy, while keeping only one additional copy of attention-related parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “Previous works on multilingual NMT typically trained models with up to 7 languages (Dong et al., 2015; Firat et al., 2016b; Ha et al., 2016; Johnson et al., 2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “When pivots are not easy to obtain, Firat et al. (2016a); Lee et al. (2016); Johnson et al. (2016) have shown that the structure of NMT is suitable for multilingual machine translation.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Early multilingual NMT models utilized separate vo-cabularies for each language (Dong et al., 2015; Luong et al., 2015; Firat et al., 2016a); later ones used shared multilingual vocabularies (Sennrich et al., 2016; Ha et al., 2016c; Johnson et al., 2017) which are shared across languages.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Various forms of transfer or multitask learning with seq2seq framework also have the flavors of our algorithm (Zoph et al., 2016; Luong et al., 2015; Firat et al., 2016).”
Complete Multilingual Neural Machine Translation (2020, neutral): “Multilingual machine translation (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019), which can serve multiple language pairs with a single model, has attracted much attention.”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, methodology): “In this section, we explore two ways of leveraging available parallel data to improve zero-shot translation quality, similar in spirit to what was reported in [10].”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Previous work on zero-resource machine translation has also relied on labeled information, not from the language pair of interest but from other related language pairs (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017) or from other modalities (Nakayama & Nishida, 2017; Lee et al., 2017).”
Unsupervised Neural Machine Translation (2017, background): “It has however been shown that the use of more advanced models like a teacher-student framework can bring considerable improvements over this basic baseline (Firat et al., 2016b; Chen et al., 2017).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “…et al., 2015a; Gul-cehre et al., 2015; He et al., 2016; Gu et al., 2018; Wang et al., 2018); large sets of parallel sentences in related languages (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017; Zheng et al., 2017); cross-lingual dictionaries (Klementiev et al., 2012; Irvine and…”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Multilingual MT (Dong et al., 2015; Firat et al., 2016a,b; Wang et al., 2020) addresses the transfer of MT from high-resourced to low-resourced languages by training multilingual models for all languages at once.”
A Teacher-Student Framework for Zero-Resource Neural Machine Translation (2017, methodology): “Firat et al. (2016b) present a multi-way, multilingual model with shared attention to achieve zero-resource translation.”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “Multilingual NMT allows us to perform zeroshot translation, although the quality is not guaranteed (Firat et al., 2016b; Johnson et al., 2017).”
Paraphrasing Revisited with Neural Machine Translation (2017, methodology): “Firat et al. (2016b) propose several ways in which multiple pivot sentences can be incorporated into a NMT decoder.”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “Simple: Similar to Johnson et al. (2017) and Ha et al. (2016), and in contrast with Luong et al. (2016) and Firat et al. (2016a), it can be applied to most existing NMT systems with some minor modiﬁcation, and it is able to accommodate attention layers seamlessly.”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “As an extreme in terms of the number of supervised examples, prior works dug into * Equal contribution. translation with zero-resource (Firat et al., 2016b; Chen et al., 2017; Lample et al., 2018a,b) where the language pairs in interest do not have any parallel corpora between them.”
Rethinking embedding coupling in pre-trained language models (2020, background): “Embedding coupling is also common in neural machine translation models where it reduces model complexity (Firat et al., 2016) and saves memory (Johnson et al., 2017), in recent state-of-the-art language models (Melis et al., 2020), as well as all pre-trained models we are aware of (Devlin et al.,…”