Copied Monolingual Data Improves Low-Resource Neural Machine Translation

By Antonio Valerio Miceli Barone, Kenneth Heafield. 2017

200 citations recorded by the source corpus.

We train a neural machine translation (NMT) system to both translate source-language text and copy target-language text, thereby exploiting monolingual corpora in the target language. Speciﬁcally, we create a bitext from the monolingual text in the target language so that each source sentence is identical to the target sentence. This copied data is then mixed with the parallel corpus and the NMT system is trained like normal, with no metadata to distinguish the two input languages. Our proposed method proves to be an effective way of incorporating monolingual data into low-resource NMT. see gains of up to 1.2 BLEU over a strong baseline with back-translation. Further analysis shows that the linguis-tic phenomena behind these gains are different from and largely orthogonal to back-translation, with our copied corpus method improving accuracy on named entities and other words that should remain identical between the source and target languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tagged Back-Translation (2019, methodology): “, 2007; Kim and Rush, 2016), or source copying (Currey et al., 2017).”
Unsupervised Neural Machine Translation (2017, background): “e parallel corpora. A simple yet effective approach is to create a synthetic parallel corpus by backtranslating a monolingual corpus in the target language (Sennrich et al., 2016a). At the same time, Currey et al. (2017) showed that training an NMT system to directly copy target language text is als”
Iterative Back-Translation for Neural Machine Translation (2018, background): “Currey et al. (2017) reported similar ﬁndings for low resource conditions, showing that even poor translations can be beneﬁcial.”
A neural interlingua for multilingual machine translation (2018, methodology): “This may be due to the fact that even the smallest language pair in WMT15 has 2 million sentence pairs, which is more than 3 times larger than either the Tr-En or Ro-En pairs discussed in Currey et al. (2017).”
On the Impact of Various Types of Noise on Neural Machine Translation (2018, background): “Other work has also considered copying in NMT. Currey et al. (2017) add copied data and back-translated data to a clean parallel corpus.”
Simple Fusion: Return of the Language Model (2018, background): “The source sentences can be generated with a separate translation system (Schwenk, 2008; Sennrich et al., 2016a) (backtranslation), or simply copied over from the target side (Currey et al., 2017).”
Machine Translation into Low-resource Language Varieties (2021, methodology): “Considering a new target variety as a new domain of STD , unsupervised domain adaptation methods can be employed, such as ﬁnetuning SRC → STD models using pseudo-parallel corpora generated from monolingual corpora in target varieties (Hu et al., 2019; Currey et al., 2017).”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, background): “Our approach is also related but substantially different from the mixture of experts for diverse MT [20], iterative back-translation [11] and copied monolingual data for NMT [2]; see the Appendix for further details about these comparisons.”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, background): “32], and contrastive methods [33, 34, 35, 36]; • machine translation (MT): integration of a language model trained on monolingual data [37, 38, 39], backtranslation [40, 41, 42], synthetic data usage [43], and web-scale bitext mining [44]; • automatic speech recognition (ASR): representation learnin”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “Semi-supervised techniques can be categorized according to the way the monolingual data is utilized: Using monolingual data to generate synthetic parallel data: The simplest strategy is to create the synthetic parallel corpus (this is essentially data augmentation) either by 1) copying monolingual d”
Neural Machine Translation: A Review (2019, background): “, 2016b) or copying the target sentence over to the source side (Currey et al., 2017).”
Zero-Resource Neural Machine Translation with Monolingual Pivot Data (2019, methodology): “The ﬁrst is based on the copied corpus method of Currey et al. (2017).”
CUNI Submissions in WMT18 (2018, background): “Aside from the common back-translation (Sen-nrich et al., 2016), simple copying of target mono-lingual data back to source (Currey et al., 2017) has been also shown to improve translation quality in low-data conditions.”
Generalized Data Augmentation for Low-Resource Translation (2019, background): “, 2016) is a common practice in data augmentation, but has also been shown to be less effective in low-resource settings where it is hard to train a good back-translation model (Currey et al., 2017).”
The University of Edinburgh’s Neural MT Systems for WMT17 (2017, unknown): “More details can be found in Currey et al. (2017).”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, background): “…modelling objective (G¨ulc¸ehre et al., 2015; Sennrich et al., 2016b; Ra-machandran et al., 2017), an autoencoding objec-tive (Luong et al., 2016; Currey et al., 2017), or a round-trip objective, where the model is trained to predict monolingual (target-side) training data that has been…”
A Survey of Domain Adaptation for Neural Machine Translation (2018, methodology): “The data used can be either in-domain monolingual corpora (Zhang and Zong, 2016b; Cheng et al., 2016; Currey et al., 2017; Domhan and Hieber, 2017), synthetic corpora (Sennrich et al., 2016b; Zhang and Zong, 2016b; Park et al., 2017), or parallel copora (Chu et al., 2017; Sajjad et al., 2017; Britz…”
Multi-Modal Data Augmentation for End-to-end ASR (2018, background): “In [18], improvements in low-resource settings were obtained by simply copying the source-side (input) monolingual data to the target side (output).”
Code-Switching for Enhancing NMT with Pre-Specified Translation (2019, methodology): “On translating named entities, Currey et al. (2017) augment the training data by copying target-side sentences to the source-side, resulting in augmented training corpora where the source and the target sides contain identical sentences.”