A Teacher-Student Framework for Zero-Resource Neural Machine Translation

By Yun Chen, Yang Liu, Yong Cheng, Victor O. K. Li. 2017

156 citations recorded by the source corpus.

While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low-resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on the assumption, our method is able to train a source-to-target NMT model (“student”) without parallel corpora available guided by an existing pivot-to-target NMT model (“teacher”) on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “, 2019) or distillation through pivoting (Chen et al., 2017).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “For instance, Cheng et al. (2016); Chen et al. (2017); Lee et al. (2017); Chen et al. (2018) investigate the use of a pivot to build a translation path be-tween two languages even without any directed resource.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “The most effective approach has been to simply synthesize parallel data (Firat et al., 2016b; Chen et al., 2017, 2018b) and incorporate that into the training process.”
Complete Multilingual Neural Machine Translation (2020, neutral): “…non-English pairs either resort to bridging (pivoting) through a pivot language (Habash and Hu, 2009), or make use of synthetic parallel data (via back-translation) (Firat et al., 2016b; Chen et al., 2017) or study the problem under zero-shot settings (Johnson et al., 2017; Ha et al., 2016).”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Previous work on zero-resource machine translation has also relied on labeled information, not from the language pair of interest but from other related language pairs (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017) or from other modalities (Nakayama & Nishida, 2017; Lee et al.”
Unsupervised Neural Machine Translation (2017, background): “It has however been shown that the use of more advanced models like a teacher-student framework can bring considerable improvements over this basic baseline (Firat et al., 2016b; Chen et al., 2017).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “…et al., 2016; Gu et al., 2018; Wang et al., 2018); large sets of parallel sentences in related languages (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017; Zheng et al., 2017); cross-lingual dictionaries (Klementiev et al., 2012; Irvine and Callison-Burch, 2014, 2016); or comparable…”
A neural interlingua for multilingual machine translation (2018, methodology): “Various extensions of the pivot technique have been proposed over the years, see Utiyama and Isahara (2007), Chen et al. (2017), Miura et al. (2017), Cohn and Lapata (2007).”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “This was recently extended using a teacher-student framework (Chen et al., 2017).”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “Also our methods can easily beat (Chen et al., 2017)”
A Stable and Effective Learning Strategy for Trainable Greedy Decoding (2018, methodology): “a pseudo-parallel corpus generated from the underlying NMT model (Gao and He, 2013; Auli and Gao, 2014; Kim and Rush, 2016; Chen et al., 2017; Freitag et al., 2017; Zhang et al., 2017) for actor training.”
A Study of Multilingual Neural Machine Translation (2019, background): “…language pair, pivot translation (e.g., translation from the source language to a pivot language like English and then from the pivot language to the target language) is widely adopted in real-world systems (Johnson et al. 2017; Ha, Niehues, and Waibel 2016; Cheng et al. 2017; Chen et al. 2017).”
Unsupervised Pivot Translation for Distant Languages (2019, methodology): “Cheng et al. (2017); Chen et al. (2017) also adapt the pivot based method into neural machine translation.”
Conditional Teacher-student Learning (2019, background): “Teacher-student (T/S) learning [1, 2] has been widely applied to a variety of deep learning tasks in speech, language and image processing including model compression [1, 2], domain adaptation [3, 4, 5], small-footprint natural machine translation (NMT) [6], low-resource NMT [7], far-field automatic”
Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “In addition, zero-resource techniques which generate artificial data for the missing directions have been proposed as an alternative to zero-shot translation (Chen et al., 2018; Al-Shedivat and Parikh, 2019; Chen et al., 2017).”
Zero-Shot Cross-Lingual Abstractive Sentence Summarization through Teaching Generation and Attention (2019, methodology): “We adopt the teacher-student framework that is also applied in Chen et al. (2017), but we have significant difference to them in that we generate pseudo source while they generate pseudo target, which results in different teacher-student networks.”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “Hence, there has been a massive increase in work on MT systems that involve more than two languages (Dong et al., 2015; Firat et al., 2016a; Zoph and Knight, 2016; Cheng et al., 2017; Johnson et al., 2017; Chen et al., 2017, 2018b; Neu-big and Hu, 2018) etc .”
Autoregressive Knowledge Distillation through Imitation Learning (2020, unknown): “Subsequent works have used seqKD for non-autoregressive translation models (Gu et al., 2017; Zhou et al., 2019), low-resource settings (Chen et al., 2017), and ensemble distillation with multiple teachers (Kuncoro et al., 2016; Tan et al., 2019).”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, unknown): “Similarly, parallel data from other language pairs can be used to pre-train the network or jointly learn representations (Zoph et al., 2016; Chen et al., 2017; Nguyen and Chiang, 2017; Neubig and Hu, 2018; Gu et al., 2018a,b; Kocmi and Bojar, 2018).”