Synthetic and Natural Noise Both Break Neural Machine Translation

By Yonatan Belinkov, Yonatan Bisk. 2017

813 citations recorded by the source corpus.

Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

Subfields: Artificial Intelligence, Software, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Software

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, neutral): “Similar observations were made in machine translation (Belinkov and Bisk, 2017) and dialogue (Sankar et al.”
WinoGrande (2019, neutral): “This approach is also reminiscent of recent work in NLP on adversarial learning (Chen and Cardie, 2018; Belinkov and Bisk, 2018; Elazar and Goldberg, 2018).”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, neutral): “This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “(Belinkov and Bisk, 2017) explore the training of character-based NMT with a synthetic noise that randomly changes the order of characters in a word.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “, 2019), machine translation (Isabelle et al., 2017; Belinkov and Bisk, 2018) and language modelling (Marvin and Linzen, 2018; Ettinger, 2020).”
Shortcut learning in deep neural networks (2020, background): “Feature combination crucially depends on shortcut features like word length [59, 60, 14, 61], and consequently leads to a severe lack of robustness such as an inability to generalise to more challenging test conditions [62, 63, 64, 65].”
Robust Neural Machine Translation with Joint Textual and Phonetic Embedding (2018, background): “Despite tremendous success, NMT models are very sensitive to the noises in input sentences (Belinkov and Bisk, 2017).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “[4] Belinkov, Y.”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, neutral): “Nevertheless algorithmic approaches have been attempted, for example in text classiﬁcation (Ebrahimi et al., 2018), machine translation (Belinkov and Bisk, 2018), dialogue generation tasks (Li et al., 2017) and reading comprehension (Jia and Liang, 2017).”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “There has separately been significant work on better understanding the behavior NLP models, for example by crafting inputs that cause a misclassification (Jia & Liang, 2017; Nie et al., 2019) or diagnosing why they sometimes generate nonsensical text (Lee et al., 2018; Belinkov & Bisk, 2017).”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “Khayrallah and Koehn (2018) and Belinkov and Bisk (2018) show that neural translation models are far more sensitive to noisy parallel training data than statistical machine translation.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “on (Sakaguchi et al., 2017; Heigold et al., 2018; Belinkov and Bisk, 2018).”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, methodology): “, 2018) and stress test neural machine translation (Belinkov and Bisk, 2018).”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “Generating synthetic data via para-phrases and perturbations has been commonly used for generating adversarial examples (Jia and Liang, 2017; Iyyer et al., 2018; Belinkov and Bisk, 2018; Ribeiro et al., 2018), an orthogonal line of research.”
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020, background): “There are existing perturbation techniques meant to evaluate speciﬁc behavioral capabilities of NLP models such as logical consistency (Ribeiro et al., 2019) and robustness to noise (Belinkov and Bisk, 2018), name changes (Prabhakaran et al., 2019), or adversaries (Ribeiro et al., 2018).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Similarly, machine translation systems have been found susceptible to character-level perturbations (Ebrahimi et al., 2018a) and synthetic and natural noise (Belinkov and Bisk, 2018; Khayrallah and Koehn, 2018).”
Stress Test Evaluation for Natural Language Inference (2018, background): “Unlike work on manual construction of small adversarial evaluation sets for various NLP tasks (Levesque, 2014; Mahler et al., 2017; Staliūnaite and Bonfil, 2017; Isabelle et al., 2017; Belinkov and Bisk, 2017; Bawden et al., 2017), our work focuses on a”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “• Since NMT is very vulnerable to noisy training data, rare occurrences in the data, and the training data quality in general [4].”
Datasets: A Community Library for Natural Language Processing (2021, background): “…compete with human performance, many research projects have demonstrated that these same models are fooled when given out-of-domain examples (Koehn and Knowles, 2017), simple adversarial constructions (Belinkov and Bisk, 2018), or examples that spuriously match basic patterns (Poliak et al., 2018).”
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets (2019, background): “These challenges have been used as evidence that current systems are brittle (Belinkov and Bisk, 2018; Mudrakarta et al., 2018; Zhao et al., 2018; Glockner et al., 2018; Ebrahimi et al., 2018; Ribeiro et al., 2018, 1 Often referred to as “adversarial datasets” or “attacks”.”