MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

By Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal. 2018

105 citations recorded by the source corpus.

We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2019), tracking entity state changes (Mishra et al., 2018; Ostermann et al., 2018) or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al.”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “Science questions often require common sense, and have recently received attention (Clark et al., 2018; Mi-haylov et al., 2018; Ostermann et al., 2018); however, they also need specialized scientiﬁc knowledge.”
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (2019, background): “…commonsense reasoning, in contrast with existing machine comprehension (MRC) datasets such as SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), Narrative QA (Koˇcisk`y et al., 2018), and MCScript (Ostermann et al., 2018), where only a relatively smaller portion of the questions (e.g., 27 .”
What Makes Reading Comprehension Questions Easier? (2018, neutral): “MCScript (Ostermann et al., 2018) 11.”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “Another line of evaluation involves commonsense-enabled reading comprehension and question answering (Ostermann et al., 2018; Zhang et al., 2018; Huang et al., 2019).”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, methodology): “Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
SemEval-2020 Task 4: Commonsense Validation and Explanation (2020, background): “…with adversarial generations (Zellers et al., 2018), event validation (Wang et al., 2018), reading comprehension (Mostafazadeh et al., 2016; Ostermann et al., 2018b; Ostermann et al., 2018a), dialogue (Cui et al., 2020) and QA problems (Davis, 2016; Talmor et al., 2018; Mihaylov et al.,…”
MITRE at SemEval-2018 Task 11: Commonsense Reasoning without Commonsense Knowledge (2018, methodology): “2 Task, Data and Evaluation Machine Comprehension using Commonsense Knowledge was a shared task organized within SemEval-2018 (Ostermann et al., 2018b).”
Comparing Test Sets with Item Response Theory (2021, methodology): “8 MCScript (Ostermann et al., 2018) 14,191 2,020 3,610 Acc.”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, background): “Small 79.4 80.7 67.6 66.4 77.1 78.0 45.5 53.7 55.9 55.2 50.4 50.0 42.9 47.1 35.9 37.0 80.0 72.8 59.4 60.1 0.7 60 Base 88.2 86.8 78.1 74.7 82.5 82.4 49.8 60.4 70.3 69.0 59.0 59.4 53.0 58.5 42.4 45.7 86.9 86.3 67.8 69.2 1.5 50 Large 91.1 90.8 85.9 86.3 86.1 85.8 51.6 63.2 78.0 78.8 68.2 71.8 65.9 70.5”
Commonsense Reasoning for Natural Language Understanding: A Survey of Benchmarks, Resources, and Approaches (2019, methodology): “(A) MCScript (Ostermann et al., 2018) Did they throw away the old diaper?”
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge (2018, methodology): “For our shared task, we use the MCScript data set (Ostermann et al., 2018).”