Semi-Supervised Learning for Neural Machine Translation

By Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu. 2016

261 citations recorded by the source corpus.

While end-to-end neural machine translation (NMT) has made remarkable progress recently, NMT systems only rely on parallel corpora for parameter estimation. Since parallel corpora are usually limited in quantity, quality, and coverage, especially for low-resource languages, it is appealing to exploit monolingual corpora to improve NMT. We propose a semi-supervised approach for training NMT models on the concatenation of labeled (parallel corpora) and unlabeled (monolingual corpora) data. The central idea is to reconstruct the monolingual corpora using an autoencoder, in which the source-to-target and target-to-source translation models serve as the encoder and decoder, respectively. Our approach can not only exploit the monolingual corpora of the target language, but also of the source language. Experiments on the Chinese-English dataset show that our approach achieves significant improvements over state-of-the-art SMT and NMT systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, background): “Back translation [55, 15] and dual learning [20, 5] can be regarded as performing data augmentation on monolingual data and have been shown to improve the performance for machine translation.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “In our experiments, we benchmark our method on machine translation, where other unsupervised methods are shown to give promising results (Sennrich et al., 2015a; Cheng et al., 2016).”
Tagged Back-Translation (2019, background): “…et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Back translation and self-training have led to significant improvements in machine translation [71, 20, 28, 14, 88, 29].”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Thanks to recent advances in deep learning (Sutskever et al., 2014; Bahdanau et al., 2015) and the availability of large-scale parallel corpora, machine translation has now reached impressive performance on several language pairs (Wu et al., 2016).”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, background): “Researchers have worked on a number of meth-ods for using monolingual data in NMT systems (Cheng et al., 2016; He et al., 2016; Ramachan-dran et al., 2016).”
A Teacher-Student Framework for Zero-Resource Neural Machine Translation (2017, methodology): “To address this problem, it is possible to construct a sub-space by either sampling (Shen et al., 2016), generating a k -best list (Cheng et al., 2016b) or mode approximation (Kim and Rush, 2016).”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, methodology): “…P (y|x), i.e., making sampled sentences x relevant to the given sentence y. Inspired by the ideas of dual learning (He et al. 2016a) and backtranslation (Cheng et al. 2016), we propose to get relevant source sentence x for a given target sentence y by sampling from a dual translation model P (x|y).”
The Neural Noisy Channel (2016, background): “Cheng et al. (2016) leverages the abundant monolingual data by doing multitask learning with an autoencoding objective.”
Exploiting Source-side Monolingual Data in Neural Machine Translation (2016, methodology): “Most of the existing methods mainly focus on designing better alignment mechanisms (attention model) for the decoder network (Cheng et al., 2016a; Luong et al., 2015b; Cohn et al., 2016; Feng et al., 2016; Tu et al., 2016; Mi et al., 2016a; Mi et al., 2016b), better objective functions for BLEU…”
Listening while speaking: Speech chain by deep learning (2017, background): “Another similar work in neural machine translation was introduced by Cheng et al. [34].”
Neural Machine Translation with Reconstruction (2016, methodology): “It has been shown that combination of two directional models outperforms each model alone (Liang, Taskar, and Klein 2006; Cheng et al. 2016a; Cheng et al. 2016b).”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “In addition to image classification, self-training was also successfully applied to a variety of tasks, such as semantic segmentation [7], text classification [96, 77], machine translation [137, 62, 23, 63] and learning from noisy data [152].”
APE at Scale and Its Implications on MT Evaluation Biases (2019, methodology): “In a similar formulation, Cheng et al. (2016) jointly learn a translation system with a round-trip autoencoder.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “…methods for NLP including self-training (Yarowsky, 1995; Zhang and Zong, 2016; He et al., 2020; Lin et al., 2020), generative methods (Xu et al., 2017; Yang et al., 2017; Kingma et al., 2014; Cheng et al., 2016), and co-training (Blum and Mitchell, 1998; Clark et al., 2018; Cai and Lapata, 2019).”
Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models (2019, methodology): “The approach of joint learning has been tested in the literature in other domains than NLG/NLU for tasks such machine translation (Cheng et al., 2016; He et al., 2016; Tu et al., 2017) and speech processing (Tjandra et al., 2017, 2018; Liu et al., 2018).”
Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, background): “…data is an important research topic for NMT and there are plenty of works studying this problem (Gulcehre et al., 2015, 2017; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Zhang et al., 2018; Cheng et al., 2016; Wang et al., 2018; Chinea-Rios et al., 2017; Wu et al., 2018).”
Dictionary-based Data Augmentation for Cross-Domain Neural Machine Translation (2020, methodology): “Cheng et al. (2016) reconstruct the monolingual corpora using an autoen-coder, in which both source and target monolingual data can be used.”