On Using Monolingual Corpora in Neural Machine Translation

By Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loïc Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, Yoshua Bengio. 2015

591 citations recorded by the source corpus.

Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to $1.96$ BLEU improvement on the low-resource language pair Turkish-English, and $1.59$ BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of $0.39$ and $0.47$ BLEU scores over the neural machine translation baselines, respectively.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, background): “In addition, we will discuss the unsupervised learning tasks considered as well as the learning process.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “Many research efforts have been spent on incorporating the monolingual corpora into machine translation, such as multi-task learning (Gulcehre et al., 2015; Zhang and Zong, 2016), back-translation (Sen-nrich et al., 2015), dual learning (He et al., 2016) and unsupervised machine translation with…”
Improving Neural Machine Translation Models with Monolingual Data (2015, methodology): “For Turkish → English, we use gradient clipping with threshold 5, following Gülçehre et al. (2015), in contrast to the threshold 1 that we use for English ↔ German, following Jean et al. (2015a).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…in low and zero resource settings, including research on back-translation (Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et…”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Perhaps most closely related to our method is the work by Gulcehre et al. (2015), who combined a language model with an already trained seq2seq model by fine-tuning additional deep output layers.”
Tagged Back-Translation (2019, methodology): “…using monolingual data for machine translation include language model fusion for both phrase-based (Brants et al., 2007; Koehn, 2009) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al.,…”
Hierarchical Neural Story Generation (2018, neutral): “Gulcehre et al. (2015) combined a trained language model with a trained seq2seq model to learn a gating function that joins them.”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “To further enhance end-to-end system’s performance, we train byte-pair encoding (BPE)-level language models and apply them to the system by shallow fusion [35,36].”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “answering tasks are extraction rather than generation, it is also not sensible to use unlabeled text to improve language modeling as in machine translation (Gulcehre et al., 2015).”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “The fusion weight [71] and the non-blank reward [83], [84] are selected by a small random exploration.”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “We end with reviewing shallow fusion [21], which we have used to incorporate language models for further gains in performance.”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We use shallow fusion with a language model (LM) [4] on the teacher network to generate better transcripts for the student network to train on.”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Another way to leverage monolingual data on the target side is to augment the decoder with a language model (Gulcehre et al., 2015).”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, background): “Language Model Integration during decoding can be implemented by extending the beam search cost [12, 4, 15]:”
Dual Learning for Machine Translation (2016, methodology): “In the ﬁrst category [2, 5], monolingual corpora in the target language are used to train a language model, which is then integrated with the MT models trained from parallel bilingual corpora to improve the translation quality.”
End-to-end attention-based large vocabulary speech recognition (2015, background): “Joint training of the ARSG and an external language model is also possible [25].”
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism (2016, background): “This is done by the recently proposed deep fusion (Gulcehre et al., 2015).”