Investigating Backtranslation in Neural Machine Translation

By Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, Peyman Passban. 2018

154 citations recorded by the source corpus.

A prerequisite for training corpus-based machine translation (MT) systems – either Statistical MT (SMT) or Neural MT (NMT) – is the availability of high-quality parallel data. This is arguably more important today than ever before, as NMT has been shown in many studies to outperform SMT, but mostly when large parallel corpora are available; in cases where data is limited, SMT can still outperform NMT. Recently researchers have shown that back-translating monolingual data can be used to create synthetic parallel corpora, which in turn can be used in combination with authentic parallel data to train a highquality NMT system. Given that large collections of new parallel text become available only quite rarely, backtranslation has become the norm when building state-of-the-art NMT systems, especially in resource-poor scenarios. However, we assert that there are many unknown factors regarding the actual effects of back-translated data on the translation capabilities of an NMT model. Accordingly, in this work we investigate how using back-translated data as a training corpus – both as a separate standalone dataset as well as combined with human-generated parallel data – affects the performance of an NMT model. We use incrementally larger amounts of back-translated data to train a range of NMT systems for German-to-English, and analyse the resulting translation performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tagged Back-Translation (2019, background): “, 2015, 2017), backtranslation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al.”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, methodology): “The ﬁrst category is based on back-translation (Sennrich et al., 2016b; Poncelas et al., 2018), which utilizes monolingual data to augment a parallel training corpus.”
Text Repair Model for Neural Machine Translation (2019, background): “Iterative Back-translation Iterative back-translation (Zhang et al., 2018; Cotterell and Kreutzer, 2018; Vu Cong Duy Hoang and Cohn, 2018) is a joint training algorithm to enhance the effect of monolingual source and target data by iteratively boosting the source-to-target and target-to-source trans”
Learning Credible Deep Neural Networks with Rationale Regularization (2019, methodology): “• Data Augmentation : Back translation is an effective data augmentation method to boost model performance, e.g., machine translation [39], [40].”
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation (2018, background): “Decreases in performance with more synthetic than real data is also inline with ﬁndings of Poncelas et al. (2018).”
A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation (2020, background): “It has been proved e ﬀ ective by many works [27–30].”
The University of Maryland’s Kazakh-English Neural Machine Translation System at WMT19 (2019, methodology): “We observe that in both cases ( Back-translation , Empty ) the BLEU score of the system trained on the augmented data fails to outperform the Baseline-Transfer performance, possibly due to the fact that the real Kazakh data have been “overwhelmed” by the auxiliary ones (Poncelas et al., 2018).”
APE at Scale and Its Implications on MT Evaluation Biases (2019, background): “Back-translation Back-translation (Sennrich et al., 2016b; Ponce-las et al., 2018) augments relatively scarce parallel data with plentiful monolingual data, allowing one to train source-to-target (S2T) models with the help of target-to-source (T2S) models.”
Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, background): “Poncelas et al. (2018) study on how using the back translated data as a training corpus (separately usage or combined with bilingual data) affects the performance of an NMT model.”
CUNI Submission for Low-Resource Languages in WMT News 2019 (2019, background): “It seems that using only synthetic data generates the best performance, and whenever we increase the authentic part, the performance slowly decreases, contrary to the Poncelas et al. (2018).”
CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT (2020, methodology): “Although backtranslation has been successfully used to augment text data in a variety of NLP tasks (Yu et al., 2018; Poncelas et al., 2018), to our knowledge, the technique is yet to be applied to a medical report extraction task.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, background): “tracts much attention as its success in MT (Poncelas et al., 2018; Edunov et al., 2018).”
Neural Machine Translation: A Review (2019, unknown): “A major limitation of back-translation is that the amount of synthetic data has to be balanced with the amount of real parallel data (Sennrich et al., 2016b, 2016a; Poncelas et al., 2018).”
Simple Fusion: Return of the Language Model (2018, background): “Since data augmentation methods rely on some balance between real and synthetic data (Sennrich et al., 2016a; Currey et al., 2017; Poncelas et al., 2018), they can often only use a small fraction of the available monolingual data.”