How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

By Divyansh Kaushik, Zachary Chase Lipton. 2018

239 citations recorded by the source corpus.

Many recent papers address reading comprehension, where examples consist of (question, passage, answer) tuples. Presumably, a model must combine information from both questions and passages to predict corresponding answers. However, despite intense interest in the topic, with hundreds of published papers vying for leaderboard dominance, basic questions about the difficulty of many popular benchmarks remain unanswered. In this paper, we establish sensible baselines for the bAbI, SQuAD, CBT, CNN, and Who-did-What datasets, finding that question- and passage-only models often perform surprisingly well. On 14 out of 20 bAbI tasks, passage-only models achieve greater than 50% accuracy, sometimes matching the full model. Interestingly, while CBT provides 20-sentence passages, only the last is needed for accurate prediction. By comparison, SQuAD and CNN appear better-constructed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “In question answering, Kaushik and Lipton (2018) showed that question-and passage-only models can perform surprisingly well, while Jia and Liang (2017) added adversarially constructed sentences to passages to cause a drastic drop in performance.”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “Some works (Chen et al., 2016; Sugawara et al., 2018; Kaushik and Lipton, 2018) classify the reasoning skills required to answer the questions into the following types: (1) Exact matching/Paraphrasing; (2) Summary; (3) Logic reasoning; (4) Utilizing external knowledge; (5) Numerical reasoning.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “A recent line of work (Gururangan et al., 2018; Kaushik and Lipton, 2018) has identiﬁed that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that…”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “An increasing number of analyses (Jia and Liang, 2017; Rajpurkar et al., 2018; Kaushik and Lipton, 2018) have revealed that a large portion of questions in these datasets can be answered by simply matching the patterns between the question and the answer sentence in the passage.”
Shortcut learning in deep neural networks (2020, background): “Shortcut learning starts from various dataset biases such as annotation artefacts [55, 56, 57, 58].”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “Furthermore, unlike cloze-based datasets, SQuAD is robust to passage/questiononly adversarial attacks (Kaushik and Lipton, 2018).”
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, background): “Across myriad tasks, researchers have demonstrated that models tend to rely on spurious associations (Poliak et al., 2018; Gururangan et al., 2018; Kaushik & Lipton, 2018; Kiritchenko & Mohammad, 2018).”
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2020, methodology): “…of the CNN-DM dataset (Hermann et al., 2015), Sugawara et al. (2020) perform an analysis of machine comprehension dataset dif-ﬁculty, Kaushik and Lipton (2018) analyse the dif-ﬁculty of various machine reading datasets, and Manjunatha et al. (2018) show that visual question answering models…”
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (2019, background): “In light of recent work exposing predictive artifacts in crowdsourced NLP datasets (Gururangan et al., 2018; Kaushik and Lipton, 2018, inter alia ), we estimate the effect of predictive artifacts by training BERT QA and XLNet QA to predict a single start and end index given only the passage as…”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “, 2015) and has been shown to be less susceptible to annotation artifacts than other popular tasks (Kaushik and Lipton, 2018).”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Recent examples include hypothesis-only baselines for natural language inference (Gururangan et al., 2018) and passage-only baselines for reading comprehension (Kaushik and Lipton, 2018).”
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering (2018, background): “, 2016) or manually annotated data sets (Gururangan et al., 2018; Kaushik and Lipton, 2018).”
Evaluating NLP Models via Contrast Sets (2020, background): “Nevertheless, the presence of these gaps is well-documented (Gururangan et al., 2018; Poliak et al., 2018; Min et al., 2019; Kaushik and Lipton, 2018).”
Unsupervised Question Answering by Cloze Translation (2019, methodology): “We do, however, use the SQuAD development set to assess which model components are Sliding window (Rajpurkar et al., 2016) 13.0 20.0 Context-only (Kaushik and Lipton, 2018) 10.9 14.8 Random (Rajpurkar et al., 2016) 1.3 4.3 Fully Supervised Models EM F1 BERT-Large (Devlin et al., 2018) 84.1 90.9…”
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020, background): “Annotation De-biasing As has been suggested in previous papers (Kaushik and Lipton, 2018; Chen and Durrett, 2019; Clark et al., 2019), the existing benchmarks on multi-hop reasoning question answering have annotation biases, which makes designing multi-hop models unnecessary.”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, background): “…can prevent users from trusting model predictions (Kim, 2015; Ribeiro et al., 2016), makes it hard to detect model or data deﬁciencies (Gururangan et al., 2018; Kaushik and Lipton, 2018) or verify that a model is fair and does not exhibit harmful biases (Sun et al., 2019; Holstein et al., 2019).”