Improving Neural Machine Translation Models with Monolingual Data

By Rico Sennrich, Barry Haddow, Alexandra Birch. 2015

2,976 citations recorded by the source corpus.

Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Target-side monolingual data plays an important role in boosting fluency for phrase-based statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with separately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolingual data without changing the neural network architecture. By pairing monolingual training data with an automatic back-translation, we can treat it as additional parallel training data, and we obtain substantial improvements on the WMT 15 task English German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish->English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives substantial improvements for the IWSLT 15 task English->German.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “This idea has previously been applied to sequence-level knowledge distillation by Kim and Rush (2016) and makes the training procedure similar to back-translation (Sennrich et al., 2016).”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “While the concept of backtranslation has been introduced before, it is often used to improve either the same translation task Sennrich et al. (2016) or instrinsic paraphrase evaluations Wieting et al. (2017); Mallinson et al. (2017).”
Universal Language Model Fine-tuning for Text Classification (2018, background): “MT domains (Sennrich et al., 2015) but has been shown to fail between unrelated ones (Mou et al.”
Language Models are Few-Shot Learners (2020, background): “Existing unsupervised machine translation approaches often combine pretraining on a pair of monolingual datasets with back-translation [SHB15] to bridge the two languages in a controlled way.”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “, 2020); the latter commonly relies on back-translation (Sennrich et al., 2016), requiring large amounts of parallel data.”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “When used as an augmentation method, backtranslation [16, 17] refers to the procedure of translating an existing example x in language A into another language B and then translating it back into A to obtain an augmented example x̂.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “The ﬁrst attempts at tackling this problem exploited the availability of monolingual corpora (Gulcehre * Equal contribution. et al., 2015; Sennrich et al., 2015; Zhang and Zong, 2016).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “In addition to the growing number of models, ﬁne-tuning requires very careful hyper-parameter tuning (eg. learning rate, regularization knobs etc.) during adaptation, and is prone to rapid over-ﬁtting (Sennrich et al., 2016; Miceli Barone et al., 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…work on incorporating monolingual data to improve translation performance in low and zero resource settings, including research on back-translation (Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai…”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Experimental settings: We use subword units (Sennrich et al., 2015b) with 31500 merges, resulting in a vocabulary size of about 32000.”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Backtranslation (Sennrich et al., 2016; Lample et al., 2018a;b) is a common approach that uses the target-to-source model to generate noisy sources and then trains the source-to-target model to reconstruct the targets (also related to cycle-consistency (Zhu et al., 2017; Hoffman et al., 2018) and…”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “This method of back-translation to create additional parallel data has been shown to be effective in improving the performance of neural machine translation systems (Sennrich et al., 2016; Hoang et al., 2018; Edunov et al., 2018).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “We learn shared BPE (Sennrich et al., 2016b) merge operations (with 32k merge ops) across all the training data and added monolingual data as a supplement (limit to 1M sentences).”
Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “Do the normal back translation (Sennrich et al., 2016a; He et al., 2016) using P1 and P2.”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “Given the small amount of EN–KK parallel data (see Table 1) and the large amount of EN–RU and KK–RU datasets, we introduced RU as a pivot language, using backtranslation (Sennrich et al., 2015) to derive bigger datasets where the source side is synthetic.”
Tagged Back-Translation (2019, methodology): “That the tagged-decode B LEU is still quite reasonable tells us that the model has not simply learned to ignore the source sentence when it encounters the input tag, suggesting that the p ( y | BT ( x )) signal is still useful to the model, as Sennrich et al. (2016) also demonstrated.”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Back-translation (Sennrich et al., 2016b) is an essential method to integrate the target side monolingual synthetic knowledge when building a stateof-the-art NMT system.”