A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

By Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James F. Allen. 2016

827 citations recorded by the source corpus.

Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the 'Story Cloze Test'. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of ~50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Next we study the variance of few-shot learning performance on the StoryCloze, Natural Questions, and Web Questions benchmarks presented in Section 6.1.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (extracted from Common Crawl), ﬁction (extracted from Sto-ryCloze (Mostafazadeh et al., 2016) and CBT (Hill et al., 2015)), formal spoken text (excerpted from court and presidential debate ”
Story Ending Prediction by Transferable BERT (2019, neutral): “Story ending prediction, also known as the Story Cloze Test (SCT) [Mostafazadeh et al. , 2016], is an open task for evaluating story comprehension.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “Commonsense Reasoning ROCStories Corpus (Mostafazadeh et al., 2016), SWAG (Zellers et al., 2018), and The Winograd Schema Challenge (WSC) (Levesque et al., 2011) are related Re Co RD : (1) ROCStories assesses commonsense reasoning in story understanding by choosing the correct story ending from…”
Annotation Artifacts in Natural Language Inference Data (2018, background): “Schwartz et al. (2017) and Cai et al. (2017) showed that certain biases are prevalent in the ROC stories cloze task (Mostafazadeh et al., 2016), which allow models trained on the endings alone, and not the story preﬁx, to yield state-of-the-art results.”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “rom the given text, is needed to solve the problem, as illustrated in the examples in Table1. Comparing with other commonsense reasoning tasks, such as COPA (Roemmele et al., 2011), Story Cloze Test (Mostafazadeh et al., 2016), Event2Mind (Rashkin et al.,2018), SWAG (Zellers et al.,2018), ReCoRD (Zh”
Improving Question Answering with External Knowledge (2019, background): “A growing number of studies concentrate on the construction of multiple-choice machine reading comprehension (Mostafazadeh et al., 2016; Lai et al., 2017; Khashabi et al., 2018; Ostermann et al., 2018; Sun et al., 2019) or question answering tasks (Clark et al., 2018; Mihaylov et al., 2018).”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “We further conduct experiments on four representative multi-choice MRC datasets from different domains (i.e., ROCStories (Mostafazadeh et al. 2016), SemEval-2018 Task 11 (Ostermann et al. 2018), MCTest (Richardson, Burges, and Renshaw 2013), COIN Shared Task 1 (Ostermann et al. 2018)) and achieve th”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “• Fine-tuning: Although supervised fine-tuning of pre-trained LMs on task specific training data, 8StoryCloze does not have a training set, so we follow Brown et al. (2020) and sample few-shot examples from the development set instead.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…Questions (WebQS) (Berant et al., 2013) Cloze and Completion Tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016) Winograd-Style Tasks: Winograd (Levesque et al., 2012), WinoGrande (Sakaguchi et al., 2020) Common Sense Reasoning: PIQA…”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “StoryCloze (Mostafazadeh et al., 2016) • Sentiment analysis is a classic NLP task aims to understand whether a piece of text is positive or negative.”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “The Story Cloze Test (also referred to as ROC Stories) pits ground-truth endings to stories against implausible false ones (Mostafazadeh et al., 2016).”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, methodology): “The prompts are either premise sentences taken from MultiNLI’s fiction genre (Williams et al., 2018) or 2–3 sentence story openings taken from examples in ROCStories (Mostafazadeh et al., 2016).”
Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, background): “Many more generation tasks such as summarization (Narayan et al., 2018) and story generation (Mostafazadeh et al., 2016) lie on this spectrum.”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “It is only recently that we see a sudden increase in research interest toward commonsense AI, with an influx of new benchmarks and models (Mostafazadeh et al. 2016; Talmor et al. 2019; Sakaguchi et al. 2020).”
Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019, methodology): “We study transfer of natural language explanations from the CQA to SWAG (Zellers et al., 2018) and Story Cloze Test (Mostafazadeh et al., 2016).”
Hypothesis Only Baselines in Natural Language Inference (2018, methodology): “In JOCI, context (premise) sentences were taken from existing NLU datasets: SNLI, ROC Stories (Mostafazadeh et al., 2016), and COPA (Roemmele et al., 2011).”
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge (2016, background): “ConceptNet Numberbatch chose the correct ending 59.4% of the time, which is in fact slightly better than any results reported by Mostafazadeh et al. (2016), including neural nets trained on the task.”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “• Story generation: Last sentence generation for ROC stories (Mostafazadeh et al., 2016) consisting of 96,198 examples of partially written four-sentence stories as input, and a single sentence which completes the story as the target.”