Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

By Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2019

512 citations recorded by the source corpus.

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people’s everyday narratives, asking such questions as “what might be the possible reason of ...?", or “what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Measuring Massive Multitask Language Understanding (2020, background): “While these benchmarks evaluate linguistic skills more than overall language understanding, an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019).”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “CosmosQA (Huang et al., 2019) 2.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “This yields the following datasets: CosmosQA (Huang et al., 2019), DROP (Dua et al.”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, methodology): “C OSMOS QA (Huang et al. 2019) asks commonsense reading comprehension questions about everyday narratives.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “…The scalability and flexibility of crowdsourcing has enabled the creation of foundational NLP benchmarks across a wide range of sub-problems, and made it the dominant paradigm for data collection (Mihaylov et al., 2018; Rajpurkar et al., 2016; Huang et al., 2019; Talmor et al., 2019, i.a.).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “Cosmos QA (Huang et al., 2019) is a commonsense reading comprehension data set which requires reasoning over larger text passages.”
Measuring Coding Challenge Competence With APPS (2021, background): “, 2019a), commonsense reasoning (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019), logical deduction (Liu et al.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “, 2019a), and general commonsense question answering (Talmor et al., 2019; Huang et al., 2019; Wang et al., 2019a, 2020).”
ConTinTin: Continual Learning from Task Instructions (2022, methodology): “I NSTRUCTIONS was constructed in the following pipeline: Mishra et al. (2021) ﬁrst collected some popular NLP benchmarks (e.g., Cos-mosQA (Huang et al., 2019), Quoref (Dasigi et al., 2019), Winogrande (Sakaguchi et al., 2020), etc.) with their crowdsourcing instructions through engaging with their a”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “As a rough example, multiple-choice reading-comprehension question answering is a task, which the Cosmos benchmark (Huang et al., 2019) attempts to test using an accuracy metric over a speciﬁc sample of passages and questions from the English personal narrative domain.”
Explaining Question Answering Models through Text Generation (2020, background): “Several recent works learned to generate answers or explanations (McCann et al., 2018; Camburu et al., 2018; Rajani et al., 2019; Huang et al., 2019) using the standard supervised sequence-to-sequence ob-jective (Sutskever et al., 2014).”