A Challenge Set Approach to Evaluating Machine Translation

By Pierre Isabelle, Colin Cherry, George F. Foster. 2017

184 citations recorded by the source corpus.

Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system’s capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “, 2019), machine translation (Isabelle et al., 2017; Belinkov and Bisk, 2018) and language modelling (Marvin and Linzen, 2018; Ettinger, 2020).”
Evaluating Gender Bias in Machine Translation (2019, background): “These include challenge sets for language-speciﬁc idiosyncrasies (Isabelle et al., 2017), discourse phenomena (Bawden et al., 2018), pronoun translation (M¨uller et al., 2018; Webster et al., 2018), or coreference and multiword expressions (Burchardt et al., 2017).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Behavioral analyses, on the other hand, aim to assess a model’s behavior by its performance on constructed examples (e.g., Isabelle et al., 2017; Naik et al., 2018), or by visualizing important input features via saliency methods (e.g., Li et al., 2016; Murdoch et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Thus a few studies report human evaluation on their challenge sets, such as in MT (Isabelle et al., 2017; Burchardt et al., 2017).”
Stress Test Evaluation for Natural Language Inference (2018, background): “Unlike work on manual construction of small adversarial evaluation sets for various NLP tasks (Levesque, 2014; Mahler et al., 2017; Staliūnaite and Bonfil, 2017; Isabelle et al., 2017; Belinkov and Bisk, 2017; Bawden et al., 2017), our work focuses on a more exhaustive large-scale evaluation for NLI”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “While recent advances have reported near human-level performance on several language pairs using neural approaches (Wu et al., 2016; Hassan et al., 2018), other studies have highlighted several open challenges (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “The idea of creating challenging contrastive evaluation sets has a long history (Levesque et al., 2011; Ettinger et al., 2017; Glockner et al., 2018; Naik et al., 2018; Isabelle et al., 2017).”
Analyzing Uncertainty in Neural Machine Translation (2018, background): “Isabelle et al. (2017) propose a new evaluation benchmark to test whether models can capture important linguistic properties.”
Evaluating NLP Models via Contrast Sets (2020, background): “Challenge sets To probe a model’s ability to handle specific types of phenomena, recent work develops challenge sets (Glockner et al., 2018; Naik et al., 2018; Isabelle et al., 2017).”
Non-Parametric Adaptation for Neural Machine Translation (2019, background): “…translation quality NMT has shown some glaring weaknesses, including idiom processing, and rare word or phrase translation (Koehn and Knowles, 2017; Isabelle et al., 2017; Lee et al., 2018) - tasks that should be easy if the model could retain learned information from individual training examples.”
Evaluating the morphological competence of Machine Translation Systems (2017, methodology): “…to better diagnose neural models, notably by Linzen et al. (2016); Sennrich (2017), who focus respectively on the syntactic competence of Neural Language Models (NLMs) or of NMT; and by Isabelle et al. (2017); Burchardt et al. (2017), who resuscitate an old tradition of designing test suites.”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, background): “Isabelle et al. (2017) focus on specially constructed challenging examples in order to analyse differences between systems.”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “While neural machine translations are more fluent that phrase-based machine translation, it has long been observed that even high-resource models can struggle to generate faithful translations that are also syntactically correct (Isabelle et al., 2017) and the problem intensifies for longer sentence”
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation (2019, neutral): “…current NMT systems (Sutskever et al., 2014; Cho et al., 2014a; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are known to easily over-ﬁt and result in an inferior performance when the training data is limited (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017).”
Evaluating adversarial attacks against multiple fact verification systems (2019, background): “Manual construction informed by expert knowledge: Instances that exploit world knowledge, semantics, pragmatics, morphology and syntactic variation have been written and compiled into challenge datasets for Machine Translation (Isabelle et al., 2017), Sentiment Analysis (Mahler et al., 2017; Stali¯u”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2020, background): “2) External challenge tests (Naik et al., 2018; Isabelle et al., 2017; Glockner et al., 2018; Ravichander et al., 2019; McCoy et al., 2019), where examples are constructed, either through automatic methods or by experts, exercising a specific phenomenon in isolation.”
Sometimes We Want Ungrammatical Translations (2021, background): “Second, recall that many languages, including English, use word order to encode argument structure information (cf. Isabelle et al. 2017): while “the dog bit the man” might be more frequent compared to “the man bit the dog”, the latter has a very clear meaning that we may wish to preserve in some…”
Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020, unknown): “The absence of evidence for syntactic transfer across languages is consistent with the results of Dhar and Bisazza (2020), who likewise found no evidence of transfer in an LSTM LM trained on two closely related languages (French and Italian).”
Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness (2018, methodology): “…the robustness of NLP models both via controlled experiments to complement the information from the test set accuracy and test abilities of the models (Isabelle et al., 2017; B. Hashemi and Hwa, 2016; White et al., 2017) and via adversarial instances to expose weak- nesses (Jia and Liang, 2017).”