Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation
By Marzieh Fadaee, Christof Monz. 2018
113 citations recorded by the source corpus.
Neural Machine Translation has achieved state-of-the-art performance for several language pairs using a combination of parallel and synthetic data. Synthetic data is often generated by back-translating sentences randomly sampled from monolingual data using a reverse translation model. While back-translation has been shown to be very effective in many cases, it is not entirely clear why. In this work, we explore different aspects of back-translation, and show that words with high prediction loss during training benefit most from the addition of synthetic data. We introduce several variations of sampling strategies targeting difficult-to-predict words using prediction losses and frequencies of words. In addition, we also target the contexts of difficult words and sample sentences that are similar in context. Experimental results for the WMT news translation task show that our method improves translation quality by up to 1.7 and 1.2 Bleu points over back-translation using random sampling for German-English and English-German, respectively.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- David Grangier
- Isaac Caswell
- Ciprian Chelba
- Ravi Shekhar
- Surangika Ranathunga
- Mehreen Alam
- E. Lee
- Erhong Yang
In Computer Vision and Pattern Recognition
- Surangika Ranathunga
- Ravi Shekhar
- Mehreen Alam
- E. Lee
- Ciprian Chelba
- Erhong Yang
- Rishemjit Kaur
- Liner Yang
In Natural Language Processing
- Isaac Caswell
- David Grangier
- Ciprian Chelba
- Surangika Ranathunga
- Ravi Shekhar
- Mehreen Alam
- E. Lee
- Erhong Yang
Citations in the corpus
- Tagged Back-Translation (2019, background): “Fadaee and Monz (2018) find that BT ben-efits difficult-to-translate words the most, and select from the back-translated corpus by oversam-pling words with high prediction loss.”
- Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Another factor is the unavailability of sufficient monolingual data, which is again a practical problem faced by LRLs. Fadaee and Monz [53] showed that even 80k synthetic parallel sentences is not enough to obtain any meaningful results.”
- Controllable data synthesis method for grammatical error correction (2019, methodology): “We choose the test set from the CoNLL 2013 shared task [3] as our development set and report MaxMatch(M2) score [32] on the CoNLL-2014 shared task [4] test set.”