Stress Test Evaluation for Natural Language Inference

By Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn Penstein Rosé, Graham Neubig. 2018

420 citations recorded by the source corpus.

Natural language inference (NLI) is the task of determining if a natural language hypothesis can be inferred from a given premise in a justifiable manner. NLI was proposed as a benchmark task for natural language understanding. Existing models perform well at standard datasets for NLI, achieving impressive results across different genres of text. However, the extent to which these models understand the semantic content of sentences is unclear. In this work, we propose an evaluation methodology consisting of automatically constructed “stress tests” that allow us to examine whether systems have the ability to make real inferential decisions. Our evaluation of six sentence-encoder models on these stress tests reveals strengths and weaknesses of these models with respect to challenging linguistic phenomena, and suggests important directions for future work in this area.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “OOD robustness benchmarks like ours can stress test a model’s dependence on artifacts (Liu et al., 2019a; Feng et al., 2019; Naik et al., 2018).”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “, 2018) test set, which consists of examples that hypothesis-only models label incorrectly, and the NLI stress tests (Naik et al., 2018), in which sentences containing antonyms pairs, negations, high word overlap, i.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, neutral): “Much recent work has made similar observations about the limitations of existing pretrained models (Jia and Liang, 2017; Naik et al., 2018; McCoy and Linzen, 2019; McCoy et al., 2019; Liu et al., 2019a,b).”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “Naik et al. (2018) also ﬁnd evidence that NLI models use a lexical overlap heuristic, but our approach is substantially different from theirs.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “…the sets of test cases in H ATE C HECK have been used for model evaluation across a wide range of NLP tasks, such as natural language inference (Naik et al., 2018; McCoy et al., 2019), machine translation (Isabelle et al., 2017; Belinkov and Bisk, 2018) and language modelling (Marvin and…”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “While NLI stress tests have been created to mitigate these issues (Naik et al., 2018), constructing a representative NLI dataset remains an open area of research.”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “…model limitations in NLI stress tests: Liu et al. (2019a) observed catastrophic forgetting when fine-tuning DA (Parikh et al., 2016) and ESIM (Chen et al., 2016) models trained on MultiNLI (Williams et al., 2018) with the numerical-reasoning and antonym stress tests released by Naik et al. (2018).”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “We connect our results to previous work that reported instability in performance on stress tests designed to diagnose shortcut learning in Natural Language Inference tasks (McCoy et al., 2019b; Naik et al., 2018).”
Flexible Generation of Natural Language Deductions (2021, background): “…have saturated RTE benchmarks (Bowman et al., 2015; Williams et al., 2018) largely through surface-level heuristics (Gururangan et al., 2018; Poliak et al., 2018); hill-climbing on these benchmarks has failed to yield robust models (Naik et al., 2018) or systems capable of more complex reasoning.”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “heuristic on which NLI models have been shown to rely (Dasgupta et al., 2018; Naik et al., 2018).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “m-polyjuice improves accuracy on contrast and challenge sets (Kim et al., 2019; Naik et al., 2018; Glockner et al., 2018; Wang et al., 2019); it exhibits comparable (or better) gains than m-CAD (manual counterfactuals) with less implementation and annotation effort.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Behavioral analyses, on the other hand, aim to assess a model’s behavior by its performance on constructed examples (e.g., Isabelle et al., 2017; Naik et al., 2018), or by visualizing important input features via saliency methods (e.g., Li et al., 2016; Murdoch et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Some authors emphasize their wish to test systems on extreme or difﬁcult cases, “beyond normal operational capacity” (Naik et al., 2018).”
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020, background): “While many challenge sets focus on extreme or di ﬃ cult cases (Naik et al., 2018), MFTs also focus on what should be easy cases given a capability, uncovering severe bugs.”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “…the myriad stress tests for social (Rudinger et al., 2018; May et al., 2019; Nangia et al., 2020) and/or linguistic competencies (Geiger et al., 2018; Naik et al., 2018; Glockner et al., 2018; White et al., 2018; Warstadt et al., 2019; Gauthier et al., 2020; Hossain et al., 2020; Jeretic et al.,…”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “However, note that this problem is not unique to contrast sets—similar issues hold for the original test set as well as adversarial test sets (Jia and Liang, 2017), challenge sets (Naik et al., 2018), and input perturbations (Ribeiro et al., 2018a; Feng et al., 2018).”
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets (2019, methodology): “For instance, Naik et al. (2018) generated natural language inference challenge data by applying simple textual transformations to existing examples from MultiNLI (Williams et al., 2018) and SNLI (Bowman et al., 2015).”
Adversarial Filters of Dataset Biases (2020, methodology): “We evaluate on four such out-of-distribution datasets: HANS (McCoy et al., 2019b), NLI Diagnostics (Wang et al., 2018a), Stress tests (Naik et al., 2018) and Adversarial-NLI (Nie et al., 2019), see Appendix § A.3 for details.”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “While this can be helpful when building diagnostic datasets that focus on speciﬁc types of model failure (Cooper et al., 1996; Naik et al., 2018; Wang et al., 2019b), it is counterproductive when our goal is to build a broad-coverage benchmark dataset to set priorities and guide progress toward the…”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “Our strategy is close to that of Naik et al. (2018), but our modiﬁcations are more constrained and lexically targeted.”