Data Augmentation for Low-Resource Neural Machine Translation

By Marzieh Fadaee, Arianna Bisazza, Christof Monz. 2017

517 citations recorded by the source corpus.

The quality of a Neural Machine Translation system depends substantially on the availability of sizable parallel corpora. For low-resource language pairs this is not the case, resulting in poor translation quality. Inspired by work in computer vision, we propose a novel data augmentation approach that targets low-frequency words by generating new sentence pairs containing rare words in new, synthetically created contexts. Experimental results on simulated low-resource settings show that our method improves translation quality by up to 2.9 BLEU points over the baseline and up to 3.2 BLEU over back-translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “, 2018) and how to use backtranslation effectively for low resource languages (Fadaee et al., 2017).”
A Survey of Data Augmentation Approaches for NLP (2021, background): “We highlighted some in §3 and §4.1, e.g. (Sennrich et al., 2016; Fadaee et al., 2017; Xia et al., 2019).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “, based on token frequency or a language model) (Fadaee et al., 2017; Xie et al., 2017; Kobayashi, 2018).”
Data Augmentation by Pairing Samples for Images Classification (2018, methodology): “Data augmentation, applying a small mutation in the original training data and synthetically creating new samples, is widely used to virtually increase the amount of training data (e.g. Krizhevsky et al. (2012), Szegedy et al. (2015), Fadaee et al. (2017)).”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, background): “For instance, Fadaee et al. (2017) propose to replace words in the target sentences with rare words in the target vocabulary according to a language model, and then modify the aligned source words accordingly.”
Dynamic Data Selection for Neural Machine Translation (2017, background): “…experiments in Section 5.1, this is a challenging task; NMT systems are known to under-perform when trained on limited parallel data (Zoph et al., 2016; Fadaee et al., 2017), and do not have a separate large-scale target-side language model to compensate for smaller parallel training data.”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, methodology): “Back-translation (Sennrich et al., 2016), translational data augmentation (Fadaee et al., 2017), and noising (Xie et al., 2017) have shown improvements in BLEU measure for machine translation.”
Conditional BERT Contextual Augmentation (2018, background): “Fadaee(Fadaee et al., 2017) focused on the rare word problem in machine translation, replacing words in a source sentence with only rare words.”
On the Impact of Various Types of Noise on Neural Machine Translation (2018, background): “Fadaee et al. (2017) We classiﬁed any pairs of German and English sentences that are not translations of each other as misaligned sentences.”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “…properties of text including word replacements using a thesaurus (Zhang et al., 2015; Wei and Zou, 2019) and word embeddings (Wang and Yang, 2015; Fadaee et al., 2017; Kobayashi, 2018; Wu et al., 2019), and recently, generative models for synthesizing novel examples for text classiﬁcation and…”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, methodology): “This transformation uses a simple strategy to align values of a MR and tokens in the corresponding sentence inspired by how syn-onyms are substituted for tasks like machine translation (Fadaee et al., 2017).”
Data Augmentation for Spoken Language Understanding via Joint Variational Generation (2018, methodology): “In order to alleviate the data issues, data augmentation (DA) techniques that simply perform class-preserving transformation on data samples have been extensively used extensively (Simard, Steinkraus, and Platt 2003; Krizhevsky, Sutskever, and Hinton 2012; Fadaee, Bisazza, and Monz 2017).”
Soft Contextual Data Augmentation for Neural Machine Translation (2019, methodology): “In Xie et al. (2017), they replace the word with a placeholder token or a word sampled from the frequency distribution of vocabulary, showing that data noising is an effective regularizer for NMT. Fadaee et al. (2017) propose to replace a common word by low-frequency word in the target sentence, and”
Robust Neural Machine Translation with Doubly Adversarial Inputs (2019, background): “There are also papers augmenting a standard dataset based on the parallel corpora by dropping words (Sennrich et al., 2016a), replacing words (Wang et al., 2018), editing rare words (Fadaee et al., 2017), etc .”
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation (2018, background): “Fadaee et al. (2017) showed that targeting speciﬁc words during data augmentation improves the generation of these words in the right context.”
Data Augmentation via Dependency Tree Morphing for Low-Resource Languages (2018, background): “(2015) for text classification and Fadaee et al. (2017) for machine translation.”
Improving short text classification through global augmentation methods (2019, methodology): “As such we can use mixup with bag-of-words models, TFIDF [26], word embeddings and language models.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Such replacements can also be guided by a language model that takes context into consideration (Fadaee et al., 2017; Kobayashi, 2018).”
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations (2018, background): “Fadaee et al. (2017), focusing on the rare word problem in machine translation, replaced words in a source sentence with only rare words, which both of rightward and left-ward LSTM LMs independently predict with top-K conﬁdences.”