Adversarial Examples for Evaluating Reading Comprehension Systems

By Robin Jia, Percy Liang. 2017

1,727 citations recorded by the source corpus.

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of 75% F1 score to 36%; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to 7%. We hope our insights will motivate the development of new models that understand language more precisely.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “An adversarial evaluation for the Stanford Question Answering SQuAD is also used to demonstrate the robust of our model under adversarial attacks (Jia and Liang, 2017).”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “The results are shown in Table 6, where the F1 scores of other models are all extracted from Jia & Liang (2017).”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, background): “Similar kinds of errors have been observed when distractor sentences are added to the context (Jia and Liang, 2017).”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “…systems: when tested on cases sampled from a distribution that differs from the one they were trained on, their behavior is unpredictable and inconsistent with that of humans (Jia and Liang, 2017; McCoy et al., 2019b), and they require extensive instruction on each new task (Yogatama et al., 2019).”
Adversarial Training for Large Neural Language Models (2020, methodology): “The SQuAD v2.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “Adversarial examples can be created for NLP models by inserting phrases (Jia and Liang, 2017; Wallace et al., 2019), paraphrasing questions (Ribeiro et al.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “, 2019), Adversarial SQuAD (Jia and Liang, 2017), Lambada (Paperno et al.”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “Jia and Liang (2017)”
Probing Neural Network Comprehension of Natural Language Arguments (2019, background): “There is much recent work focusing on statistical cues in datasets in vision (Jo and Bengio, 2017) and NLP (Sanchez et al., 2018; McCoy et al., 2019; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Rajpurkar et al., 2018; Jia and Liang, 2017).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, neutral): “Much recent work has made similar observations about the limitations of existing pretrained models (Jia and Liang, 2017; Naik et al., 2018; McCoy and Linzen, 2019; McCoy et al., 2019; Liu et al., 2019a,b).”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, neutral): “This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “For instance, evidence suggests (Gururangan et al., 2018; Jia and Liang, 2017; McCoy et al., 2019) that simply asking human annotators to write candidate statements will result in examples that typically only require surface-level reasoning.”
Learning and Evaluating General Linguistic Intelligence (2019, background): “Jia & Liang (2017) show that reading comprehension models trained on SQuAD are very prone to adversarial attacks, although some adversaries are constructed by shifting the distribution of the inputs in non-trivial but meaning-preserving ways.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “An increasing number of analyses (Jia and Liang, 2017; Rajpurkar et al., 2018; Kaushik and Lipton, 2018) have revealed that a large portion of questions in these datasets can be answered by simply matching the patterns between the question and the answer sentence in the passage.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, methodology): “Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “Similarly, Jia and Liang (2017) show that adding an adversarially selected sentence to the instances in the SQuAD datasets drastically reduces the performance of many of the existing baselines.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, background): “Although systems can now perform well on these datasets, even matching human performance (Si-monite, 2018), they can be easily fooled (Jia and Liang, 2017); the degree to which they truly understand language or domain-speciﬁc concepts remains unclear.”
Annotation Artifacts in Natural Language Inference Data (2018, background): “A similar ﬁnding by Jia and Liang (2017) showed that the performance of top question-answering models trained on SQuAD (Rajpurkar et al., 2016) drops drastically by introducing simple adversarial sentences in the evidence.”
Shortcut learning in deep neural networks (2020, background): “Feature combination crucially depends on shortcut features like word length [59, 60, 14, 61], and consequently leads to a severe lack of robustness such as an inability to generalise to more challenging test conditions [62, 63, 64, 65].”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Some of these efforts augment humans (through guidelines, templates, programmatic generation of attacks, and various combinations thereof) to devise test cases that cause systems to fail [45, 46, 29, 21, 30, 55, 6, 23].”