Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

By Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, Matt Gardner. 2019

190 citations recorded by the source corpus.

Machine comprehension of texts longer than a single sentence often requires coreference resolution. However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to resolve coreference. We present a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia. Obtaining questions focused on such phenomena is challenging, because it is hard to avoid lexical cues that shortcut complex reasoning. We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues. We show that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark—the best model performance is 70.5 F1, while the estimated human performance is 93.4 F1.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “(3) Single-format experts are better at generalization only when the source and target datasets are very similar (for instance SQuAD and Quoref).”
Reasoning Like Program Executors (2022, background): “(iii) QuoRef (Dasigi et al., 2019): A Wikipedia-based multi-span selection RC benchmark with a special emphasis on coreference resolution.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “For example, the tasks in Quoref (Dasigi et al., 2019) is divided into two tasks of question generation and answer generation, mimicking its data collection process by humans.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, neutral): “Our work joins a line of previously introduced benchmarks targeting commonsense reasoning 365 [12, 13, 29, 15] as well as probing for other reasoning abilities [32, 33, 34, 35, 36, 37, 38].”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “We randomly sample 8,500 and 2,000 QA pairs from the training set for DROP and Quoref respectively.”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, methodology): “Examples of this method are realised in Build It Break It, The Language Edition (Ettinger et al., 2017), SWAG (Zellers et al., 2018), HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019), CODAH (Chen et al., 2019), Quoref (Dasigi et al., 2019) and Adversar-ialNLI (Nie et al., 2019).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “2016) QuoRef (Dasigi et al. 2019) TweetQA (Xiong et al.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “…are constructed using base-line models in the data collection process, either to ﬁlter out examples that existing models answer correctly (e.g., Dua et al., 2019; Dasigi et al., 2019) or to generate adversarial inputs (e.g., Zellers et al., 2018, 2019; Wallace et al., 2019b; Nie et al., 2019).”
Learning from Task Descriptions (2020, methodology): “We follow conventions established from previous work in the ﬁeld (Dua et al., 2019; Dasigi et al., 2019) in evaluating typical Reading Comprehension benchmarks and expand upon them, to account for novel output structures.”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “The intended task here is writing questions that require entity coreference (Dasigi et al., 2019).”
Evaluating NLP Models via Contrast Sets (2020, methodology): “Adversarial Construction of Contrast Sets Some recent datasets are constructed using baseline models in the data collection process, either to filter out examples that existing models answer correctly (e.g., Dua et al., 2019; Dasigi et al., 2019)”
Few-Shot Question Answering by Pretraining Span Selection (2021, background): “…datasets in various domains (Trischler et al., 2017; Kembhavi et al., 2017) and languages (Lewis et al., 2020; Clark et al., 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or multiple answer spans (Dua et al., 2019; Dasigi et al., 2019).”
Comparing Test Sets with Item Response Theory (2021, background): “Quoref (Dasigi et al., 2019) 19,399 1,209 1,209 3 EM 78.”
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, background): “This problem is studied very little in reading comprehension (Rajpurkar et al., 2016, 2018; Dua et al., 2019; Dasigi et al., 2019; Lin et al., 2019), and existing systems are hence brittle when handling questions in T ORQUE (Table 1).”
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (2021, methodology): “We analyzed crowdsourcing templates of 6 datasets: CosmosQA (Huang et al., 2019), DROP (Dua et al., 2019), MCTACO (Zhou et al., 2019), QASC (Khot et al., 2020), Quoref (Dasigi et al., 2019), and Wino-grande (Sakaguchi et al., 2020).”
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021, background): “First, Q ASPER includes a variety of answer types, including extractive, abstractive, yes/no, and unan-swerable questions, which means a typical span-selection BERT-based QA model (Devlin et al., 2019) is not sufﬁcient to support all these answer types.”