Systematic Error Analysis of the Stanford Question Answering Dataset

By Marc-Antoine Rondeau, Timothy J. Hazen. 2018

33 citations recorded by the source corpus.

We analyzed the outputs of multiple question answering (QA) models applied to the Stanford Question Answering Dataset (SQuAD) to identify the core challenges for QA systems on this data set. Through an iterative process, challenging aspects were hypothesized through qualitative analysis of the common error cases. A classifier was then constructed to predict whether SQuAD test examples were likely to be difficult for systems to answer based on features associated with the hypothesized aspects. The classifier’s performance was used to accept or reject each aspect as an indicator of difficulty. With this approach, we ensured that our hypotheses were systematically tested and not simply accepted based on our pre-existing biases. Our explanations are not accepted based on human evaluation of individual examples. This process also enabled us to identify the primary QA strategy learned by the models, i.e., systems determined the acceptable answer type for a question and then selected the acceptable answer span of that type containing the highest density of words present in the question within its local vicinity in the passage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jordan L. Boyd-Graber
2. Matt Gardner
3. Tongshuang Sherry Wu
4. Daniel S. Weld
5. Robin Jia
6. Mohit Bansal
7. Sameer Singh
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Jordan L. Boyd-Graber
2. Robin Jia
3. Tongshuang Sherry Wu
4. Pedro Rodriguez
5. Wee Chung Gan
6. Daniel S. Weld
7. Jeffrey Heer
8. Hwee Tou Ng

Citations in the corpus

• Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other work has offered insight by splitting the data into “easy” and “hard” sets with respect to some attribute and studying changes in model performance, but these methods do not offer a point-wise estimate of difficulty (Sugawara et al., 2018; Rondeau & Hazen, 2018; Sen & Saffari, 2020).”
• Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, background): “In contrast, some define groups that are highly specific to a particular dataset or model, such as hand-crafting factors to quantify MC instance difficulties (Rondeau and Hazen, 2018).”
• Information-Theoretic Measures of Dataset Difficulty (2021, unknown): “(Sugawara et al., 2018; Rondeau and Hazen, 2018; Sen and Saffari, 2020).”
• Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “This is a more mathematical formulation of the “easy” and “hard” dataset splits in question answering (Sugawara et al., 2018; Rondeau and Hazen, 2018; Sen and Saffari, 2020).”
• What Question Answering can Learn from Trivia Nerds (2019, unknown): “If you can find shortcuts to answer questions (Rondeau and Hazen, 2018; Kaushik and Lipton, 2018), so will a computer.”
• Improving the Robustness of Question Answering Systems to Question Paraphrasing (2019, background): “…that QA models trained on SQuAD tend to perform string matching to return an answer of an appropriate type near a region of significant word overlap between the context and the question (Jia and Liang, 2017; Rondeau and Hazen, 2018), we create a test set to exploit this weakness of the models.”