Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

By Atticus Geiger, Ignacio Cases, Lauri Karttunen, Christopher Potts. 2018

25 citations recorded by the source corpus.

Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity. To overcome this limitation, we present a method for generating data sets of multiply-quantified natural language inference (NLI) examples in which semantic complexity can be precisely characterized, and we use this method to show that a variety of common architectures for NLI inevitably fail to encode crucial information; only a model with forced lexical alignments avoids this damaging information loss.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Douwe Kiela
3. Yejin Choi
4. Luke S. Zettlemoyer
5. Samuel R. Bowman
6. Yonatan Belinkov
7. Caiming Xiong
8. Percy Liang

In Natural Language Processing

1. Mohit Bansal
2. Douwe Kiela
3. Samuel R. Bowman
4. Adina Williams
5. Yonatan Belinkov
6. Yejin Choi
7. Sebastian Riedel
8. Robin Jia

In Safety, Robustness & Interpretability

1. Douwe Kiela
2. Adina Williams
3. Jason Aaron Edward Weston
4. Emily Dinan
5. Tristan Thrush
6. Mohit Bansal
7. Robin Jia
8. Bertie Vidgen

Citations in the corpus

• Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “NLI systems can often be broken merely by performing simple lexical substitutions (Glockner et al., 2018), and struggle with quanti-fiers (Geiger et al., 2018) and certain superficial syntactic properties (McCoy et al., 2019).”
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “, 2018) and quantifiers (Geiger et al., 2018), or biases based on specific words (Sanchez et al.”
• Dynabench: Rethinking Benchmarking in NLP (2021, background): “…it difficult to pass the myriad stress tests for social (Rudinger et al., 2018; May et al., 2019; Nangia et al., 2020) and/or linguistic competencies (Geiger et al., 2018; Naik et al., 2018; Glockner et al., 2018; White et al., 2018; Warstadt et al., 2019; Gauthier et al., 2020; Hossain et al.,…”
• Probing Natural Language Inference Models through Semantic Fragments (2019, background): “…and small subsets of language, which includes work on syntactic probing (McCoy, Pavlick, and Linzen 2019; Goldberg 2019), probing basic reasoning (We-ston et al. 2015; Geiger et al. 2018; 2019) and probing other tasks (Lake and Baroni 2017; Chrupała and Alishahi 2019; Warstadt et al. 2019).”
• ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “…studies on the performance of NLI models and issues with existing NLI datasets, primarily focused on annotation artifacts (Gururangan et al., 2018; Geiger et al., 2018; Poliak et al., 2018b; Tsuchiya, 2018; Glockner et al., 2018a; Geva et al., 2019) and diagnostic datasets (McCoy et al., 2019;…”
• Robustness Gym: Unifying the NLP Evaluation Landscape (2021, background): “Therefore they decide to run the Standard Report for NLI.”