On the Impact of Various Types of Noise on Neural Machine Translation

By Huda Khayrallah, Philipp Koehn. 2018

230 citations recorded by the source corpus.

We examine how various types of noise in the parallel training data impact the quality of neural machine translation systems. We create five types of artificial noise and analyze how they degrade performance in neural and statistical machine translation. We find that neural models are generally more harmed by noise than statistical models. For one especially egregious type of noise they learn to just copy the input sentence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “” As stated in Khayrallah and Koehn (2018), neural machine translation models are more sensitive to noise than statistical machine translation models, so having high precision filter-”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Similarly, machine translation systems have been found susceptible to character-level perturbations (Ebrahimi et al., 2018a) and synthetic and natural noise (Belinkov and Bisk, 2018; Khayrallah and Koehn, 2018).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, neutral): “, 2020) and downstream task performance (Moore and Lewis, 2010; Xu and Koehn, 2017; Khayrallah and Koehn, 2018; Brown et al., 2020).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, background): “While Neural Machine Translation (NTM) has obtained breakthrough improvements in standard benchmarks, it is known to be particularly sensitive to the size and quality of the training data (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “Such noise is specially harmful for neural machine translation (Khayrallah and Koehn, 2018), so filtering it out is an essential processing step.”
Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, methodology): “In setting this task, we were motivated by our ongoing efforts to create large publicly available parallel corpora from web sources and the recognition that noisy parallel data is especially a concern for neural machine translation (Khayrallah and Koehn, 2018).”
Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings (2019, background): “formance, as neural machine translation (NMT) systems are less robust against noisy parallel data than statistical machine translation (SMT) systems (Khayrallah and Koehn, 2018).”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “copies and partial copies, as multiple studies (Ott et al., 2018; Khayrallah and Koehn, 2018) have noted that MT performance degrades substantially when systems are exposed to copies in training.”
Translationese as a Language in “Multilingual” NMT (2019, background): “produce domain-targeted translation (Kobus et al., 2016). Shu et al. (2019) use tags at training and inference time to increase the syntactic diversity of their output while maintaining translation quality; similarly, Agarwal and Carpuat (2019) and Marchi-”
The JHU Parallel Corpus Filtering Systems for WMT 2018 (2018, methodology): “Since much of the raw Paracrawl data is noisy (Khayrallah and Koehn, 2018), we also train a version where we simply use the portion of Paracrawl released for the shared task as the negative examples to train our classifier, without generating synthetic noisy data.”
Neural Machine Translation: A Review (2019, background): “Studies have shown that NMT is not robust against naturally occurring noise at training [448] and test [268, 451–453] time.”
Alibaba Submission to the WMT18 Parallel Corpus Filtering Task (2018, background): “Apart from the quantity and domain, the quality of parallel corpus is also very important in MT system training (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018).”
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021, methodology): “Following Khayrallah and Koehn (2018)—who, in contrast, study the impact of noise on MT —we control for different types of ﬁne-grained semantic divergences and different ratios of equivalent vs. divergent data.”
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? (2019, methodology): “ither the full training data nor the detail of the model is available to Bob. ParaCrawl subcorpus is only available to Alice and never to Bob. NMT is highly susceptible to noise in the training data (Khayrallah and Koehn, 2018), and Paracrawl is very noisy, so Alice applied dual conditional cross-en”
Improving Robustness of Machine Translation with Synthetic Noise (2019, background): “This fragility has been shown to extend to MT systems (Belinkov and Bisk, 2017; Khayrallah and Koehn, 2018) where both artiﬁcial and natural noise are shown to negatively affect performance.”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, background): “Neural machine translation systems being highly sensitive to noise coming from parallel data (Belinkov & Bisk, 2018; Khayrallah & Koehn, 2018), this may improve the performance in low-resource machine translation signiﬁcantly.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “In contrast, Khayrallah and Koehn (2018) examine noisy training data and focus on types of noise occurring in web-crawled corpora.”
Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection (2018, background): “Recent research has found that data noise has a bigger impact on neural machine translation (NMT) than on statistical machine translation (Khayrallah and Koehn, 2018), but learning what data quality (or noise) means in NMT and how to make NMT training robust to data noise remains an open research…”
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation (2018, background): “Khayrallah and Koehn (2018) studied the effects of noise in the training data on a translation model and discovered that NMT models are less robust to many types of noise than PBMT models.”