QASC: A Dataset for Question Answering via Sentence Composition

By Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, Ashish Sabharwal. 2019

401 citations recorded by the source corpus.

Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition (QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning and retrieval problems, however, remain unsolved as this model still lags by 20% behind human performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “We use the same IR sentences as used by the baselines: Aristo corpus for ARC and OBQA datasets (Clark et al., 2019c), and 2-step IR for QASC (Khot et al., 2019).”
Measuring Massive Multitask Language Understanding (2020, background): “Most either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al.”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “Two recent datasets (Khot et al., 2020a; Mihaylov et al., 2018) have considered questions requiring implicit facts.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “, 2018), QASC (Khot et al., 2020), Quoref (Dasigi et al.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “Concretely, we use RACE (Reading comprehension) (Lai et al., 2017), QASC (Khot et al., 2020), OpenBookQA (Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al.,…”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…et al., 2020), tweet_eval-stance_feminist (Barbieri et al., 2020), quarel (Tafjord et al., 2019a), openbookqa (Mi-haylov et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et al., 2019), ai2_arc (Clark et al., 2018), codah (Chen et al., 2019), superglue-copa (Gordon et al.,…”
Flexible Generation of Natural Language Deductions (2021, methodology): “While annotated resources already exist for certain domains (Khot et al., 2020; Hwang et al., 2021), this is not the case for most types of reasoning.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…poem_sentiment cls/sentiment analysis Sheng and Uthus 2020 proto_qa other Boratko et al. 2020 qa_srl other He et al. 2015 qasc qa/multiple-choice qa Khot et al. 2020 quail qa/multiple-choice qa Rogers et al. 2020 quarel qa/multiple-choice qa Tafjord et al. 2019a quartz-no_knowledge…”
Explaining Question Answering Models through Text Generation (2020, background): “First, in a synthetic setup, where the model learns to output the hypernym of objects; Second, on CSQA, which focuses on commonsense knowledge; and third, on zero-shot transfer to QASC, a multi-choice QA task with an emphasis on scientiﬁc knowledge.”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “finding overlapping words between two sentences QASC (Khot et al., 2020) Verification (VF) Identifying words essential for choosing correct answers.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “An intuitive way to solve more complex tasks, such as multi-hop question-answering (Yang et al., 2018; Khashabi et al., 2018; Khot et al., 2020) and numerical reasoning (Dua et al.”
Explanations for CommonsenseQA: New Dataset and Models (2021, background): “QASC (Khot et al., 2020) is a middle school level multiple-choice ScienceQA dataset.”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, methodology): “models in the domain of grade-school level science, which is considered particularly challenging with respect to background knowledge and inference (Clark, 2015; Clark et al., 2019; Khot et al., 2020).”
A Survey of Large Language Models (2023, neutral): “Open-Book QA Natural Questions [290], OpenBookQA [302], ARC [291], Web Questions [293], TriviaQA [294], PIQA [295], MS MARCO [303], QASC [304], SQuAD [305], WikiMovies [306]”
Editing Models with Task Arithmetic (2022, neutral): “[46]), summarization (MultiNews, Fabbri et al.”