DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

By Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. 2019

1,328 citations recorded by the source corpus.

Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 55k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs, as they remove the paraphrase-and-entity-typing shortcuts available in prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literatures on this dataset and show that the best systems only achieve 38.4% F1 on our generalized accuracy metric, while expert human performance is 96%. We additionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 51% F1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Language Models are Few-Shot Learners (2020, background): “On DROP [DWD + 19], a dataset testing discrete reasoning and numeracy, few-shot GPT-3 outperforms the ﬁne-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL + 19].”
PaLM: Scaling Language Modeling with Pathways (2022, background): “…Common Sense Reasoning: PIQA (Bisk et al., 2019), ARC (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018) • In-context Reading Comprehension: DROP (Dua et al., 2019), CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACE (Lai et al., 2017) •…”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “We are therefore interested in measuring performance on question answering tasks that require reasoning capabilities such as DROP (Dua et al., 2019).”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, positive): “In this part, we investigate the effect of different GNN structures on the DROP development set.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, neutral): “These tasks are TriviaQA, NQS, WebQS, SQuADv2, LAMBADA, DROP, QuAC and CoQA.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “0 -20 20 40 60 Percent of Data Clean in Dataset Percent change in performance of FLAN (accuracy, F1, or BLEU) eval on only clean data did better eval on all data (including dirty) did better ANLI R2 SQuAD v2 ANLI R1 DROP ReCoRD PIQA 25 500 75 100 Figure 9: Like GPT-3, we also measured performance on”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “Finally, DROP (Dua et al., 2019) is a challenging reading comprehension task that requires numerical reasoning.”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “• DROP (Dua et al., 2019): A large dataset for numerical reasoning over paragraphs.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “, 2018), and DROP (Dua et al., 2019).”
Break It Down: A Question Understanding Benchmark (2020, background): “, 2019; HudsonandManning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al.”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “We experiment on a subset of DROP (Dua et al., 2019) with numeric answers (67% of the entire dataset) focusing on mathematical reasoning.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.”
Natural Adversarial Examples (2019, background): “Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “, 2019), DROP (Dua et al., 2019), EssentialTerms (Khashabi et al.”
Large Language Models Can Self-Improve (2022, positive): “8% on DROP, +4 .”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “…or reasoning tasks expressed in natural language by including components explicitly designed to execute numerical or logical operations [27]–[32]; this is not the approach we take here, but presumably could be combined with more dataset-focused approaches to achieve improved…”