Exploiting Source-side Monolingual Data in Neural Machine Translation

By Jiajun Zhang, Chengqing Zong. 2016

297 citations recorded by the source corpus.

Neural Machine Translation (NMT) based on the encoder-decoder architecture has recently become a new paradigm. Researchers have proven that the target-side monolingual data can greatly enhance the decoder model of NMT. However, the source-side monolingual data is not fully explored although it should be useful to strengthen the encoder model of NMT, especially when the parallel corpus is far from sufﬁcient. In this paper, we propose two approaches to make full use of the source-side monolingual data in NMT. The ﬁrst approach employs the self-learning algorithm to generate the synthetic large-scale parallel data for NMT training. The second approach applies the multi-task learning framework using two NMTs to predict the translation and the reordered source-side monolingual sentences simultaneously. The extensive experiments demonstrate that the proposed methods obtain signiﬁcant improvements over the strong attention-based NMT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “Zhang & Zong (2016) designed a sentence reordering task for pre-training, but only for the encoder part of the encoder-decoder model.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “een low- and high-resource language pairs. Many research efforts have been spent on incorporating the monolingual corpora into machine translation, such as multi-task learning (Gulcehre et al., 2015; Zhang and Zong, 2016), back-translation (Sennrich et al., 2015), dual learning (He et al., 2016) and”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “, 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Similarly, Zhang and Zong (2016) found it useful to add an additional task of sentence reordering of sourceside monolingual data for neural machine translation.”
Revisiting Self-Training for Neural Sequence Generation (2019, background): “Moreover, Zhang & Zong (2016) did not update the decoder parameters when using pseudo parallel data noting that “ synthetic target parts may negatively inﬂuence the decoder model of NMT ”.”
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation (2019, neutral): “Many research efforts have been spent on how to utilize the monolingual data to improve the NMT system when only limited supervision is available (Gulcehre et al., 2015; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Yang et al., 2018).”
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, methodology): “Existing methods in NMT may be divided into two categories, based upon extra corpora (Sennrich et al., 2016a; Cheng et al., 2016; Zhang and Zong, 2016; Edunov et al., 2018) or original parallel corpora (Fadaee et al.”
A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation (2020, methodology): “propose a self-learning data augmentation method that translates monolingual source data to target, then combines it with origin data to train the final model [12].”
Machine Translation With Weakly Paired Bilingual Documents (2018, methodology): “Using monolingual data to boost the machine translation performance has attracted a lot of attention in the literature (Gulcehre et al., 2015; Sennrich et al., 2016a; Zhang & Zong, 2016; Wu et al., 2018; He et al., 2016), especially when the bilingual supervision is limited.”
A Study of Reinforcement Learning for Neural Machine Translation (2018, background): “Our work is also related with the research works that leverage monolingual data for improving NMT models (Zhang and Zong, 2016; Sennrich et al., 2015a; Wang et al., 2018; Xia et al., 2016; Cheng et al., 2016).”
The Source-Target Domain Mismatch Problem in Machine Translation (2019, background): “as this is in-domain with the text we would like to translate at test time. Uefﬁng (2006) proposed to improve a statistical MT system using selftraining (Yarowski, 1995), a direction later pursued by Zhang and Zong (2016) for neural MT. In our work, we consider the iterative variant proposed by He e”
Improving Domain Adaptation Translation with Domain Invariant and Specific Information (2019, methodology): “Zhang and Zong (2016) employs the self-learning algorithm to generate the synthetic large-scale parallel data for NMT training.”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, result): “Our ﬁnding is similar to Zhang and Zong [29], where the authors used source-side monolingual data to improve BLEU score.”
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning (2017, background): “Our approach further relates to Zhang and Zong (2016), who investigate multi-task learning for sequence-to-sequence models by strengthening the encoder using source-side monolingual data.”
Enhancement of Encoder and Attention Using Target Monolingual Corpora in Neural Machine Translation (2018, background): “On the other hand, the basis of source monolingual corpora, a pre-training method based on an autoencoder has been proposed to enhance the encoder (Zhang and Zong, 2016).”
Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “, 2019) by either forward translation of source monolingual data (Imamura and Sumita, 2018; Zhang and Zong, 2016) or back-translation of target monolingual data (Sennrich et al.”
Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, methodology): “Zhang and Zong (2016) propose self learning approach to generate the synthetic data for the source-side monolingual data, which is a semi-supervised method.”
Adapting Neural Machine Translation with Parallel Synthetic Data (2017, background): “Different works have tackled the inclusion of monolingual data, either in source (Zhang and Zong, 2016b) and target language (Gulcehre et al.”
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (2018, background): “Various recent works have attempted to improve NMT by scaffolding translation task on a single auxiliary task (Domhan and Hieber, 2017; Zhang and Zong, 2016; Dalvi et al., 2017).”