What Do Models Learn from Question Answering Datasets?

By Priyanka Sen, Amir Saffari. 2020

80 citations recorded by the source corpus.

While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate what models are really learning from QA datasets by evaluating BERT-based models across five popular QA datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect information in datasets, and ability to handle variations in questions. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other work has offered insight by splitting the data into “easy” and “hard” sets with respect to some attribute and studying changes in model performance, but these methods do not offer a point-wise estimate of difficulty (Sugawara et al., 2018; Rondeau & Hazen, 2018; Sen & Saffari, 2020).”
The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “Given the plethora of question-answering datasets, Yogatama et al. (2019), Talmor & Berant (2019), and Sen & Saffari (2020) evaluate the extent to which models trained on SQuAD generalize to other question-answering datasets.”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “This strategy has proven successful at identifying spurious features in the answer choices of QA datasets like SQuAD (Rajpurkar et al., 2016) and Natural Questions (Kwiatkowski et al., 2019; Sen & Saffari, 2020).”
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021, background): “Sen and Saffari (2020) show that there are shortcomings in QA datasets and evaluations by analysing their out-of-domain generalization capabilities and ability to handle question variation.”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “This is a more mathematical formulation of the “easy” and “hard” dataset splits in question answering (Sugawara et al., 2018; Rondeau and Hazen, 2018; Sen and Saffari, 2020).”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, background): “, 2020a), and QA (Ribeiro et al., 2020; Sen and Saffari, 2020)—identify negation as a challenging semantic phenomenon for models.”
Information-Theoretic Measures of Dataset Difficulty (2021, unknown): “(Sugawara et al., 2018; Rondeau and Hazen, 2018; Sen and Saffari, 2020).”