Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

By Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, Dan Roth. 2018

593 citations recorded by the source corpus.

We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. We solicit and verify questions and answers for this challenge through a 4-step crowdsourcing experiment. Our challenge dataset contains 6,500+ questions for 1000+ paragraphs across 7 different domains (elementary school science, news, travel guides, fiction stories, etc) bringing in linguistic diversity to the texts and to the questions wordings. On a subset of our dataset, we found human solvers to achieve an F1-score of 88.1%. We analyze a range of baselines, including a recent state-of-art reading comprehension system, and demonstrate the difficulty of this challenge, despite a high human performance. The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “…2012]) • Sentence completion (COPA [Roemmele et al., 2011]) • Word sense disambiguation (WIC [Pilehvar and Camacho-Collados, 2018]) • Question answering (MultiRC [Khashabi et al., 2018], ReCoRD [Zhang et al., 2018], BoolQ [Clark et al., 2019]) We use the datasets as distributed by the GLUE and…”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “The YN datasets we use are BoolQ (Clark et al., 2019a) naturally-perturbed version of this dataset, BoolQ-NP (Khashabi et al., 2020), and the binary (yes/no) subset of MultiRC (Khashabi et al., 2018).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Similarly, Khashabi et al. (2018) also provide a human performance estimate with the release of MultiRC.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “…passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mi-haylov et al., 2018; Zhang et al., 2018), or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al., 2018; Khashabi et al., 2018).”
CoQA: A Conversational Question Answering Challenge (2018, background): “, 2018), and multi-fact reasoning (Welbl et al., 2018; Khashabi et al., 2018; Talmor and Berant, 2018).”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “Most recently, Khashabi et al. (2018) proposed Mul-tiRC , a multiple-choice RC dataset that is designed to require multi-sentence reasoning and can have multiple correct answers.”
Improving Question Answering with External Knowledge (2019, background): “…how to directly or indirectly use automatically constructed knowledge bases/graphs from reference corpora (Khot et al., 2017; Kwon et al., 2018; Khashabi et al., 2018; Zhang et al., 2018) or existing external general knowledge graphs (Li and Clark, 2015; Sachan et al., 2016; Wang et al.,…”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, background): “Thus multi-choice MRC is more challenging and requires a more in-depth understanding of the given passage and question (Lai et al., 2017; Khashabi et al., 2018).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Reading comprehension datasets: BoolQ, MultiRC, OBQA.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, neutral): “SuperGLUE consists of a wide of NLU tasks, including Question Answering (Clark et al., 2019; Khashabi et al., 2018; Zhang et al., 2018), Natural Language Inference (Dagan et al.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, neutral): “, 2011), MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al.”
How many data points is a prompt worth? (2021, background): “As test data is not publicly available for SuperGLUE tasks, we set aside part of training (from 50 for CB, COPA and MultiRC to 500 for BoolQ) to use for development, and evaluate on their original validation sets.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “This in turn requires some modiﬁcations during training and inference that are discussed in Appendix A. MultiRC (Khashabi et al., 2018) is a QA task.”