LSDSem 2017 Shared Task: The Story Cloze Test

By Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, James F. Allen. 2017

195 citations recorded by the source corpus.

The LSDSem’17 shared task is the Story Cloze Test, a new evaluation for story understanding and script learning. This test provides a system with a four-sentence story and two possible endings, and the system must choose the correct ending to the story. Successful narrative understanding (getting closer to human performance of 100%) requires systems to link various levels of semantics to commonsense knowledge. A total of eight systems participated in the shared task, with a variety of approaches including.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We include the code to process the ROCStories dataset into ROCSumm, our train/test splits, the 3000 human summaries used in validation and test evaluation, Amazon Mechanical Turk templates used for data collection and evaluation.”
A Simple Method for Commonsense Reasoning (2018, methodology): “Similarly, LMs are also used to produce features for a classiﬁer in the Store Close Test 2017, giving best accuracy against other methods [29].”
Hypothesis Only Baselines in Natural Language Inference (2018, methodology): “In the Story Cloze task (Mostafazadeh et al., 2016, 2017), a model is presented with a short four-sentence narrative and asked to complete it by choosing one of two suggested concluding sentences.”
Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering…”
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (2019, background): “…common-sense, such as CommonsenseQA (Talmor et al., 2018) and Social IQa (Sap et al., 2019), and various other types of commonsense inferences (Levesque et al., 2012; Rahman and Ng, 2012; Gordon, 2016; Rashkin et al., 2018; Roemmele et al., 2011; Mostafazadeh et al., 2017; Zellers et al., 2018).”
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task (2017, methodology): “Concurrently with our work, the LSDSem 2017 shared task was held (Mostafazadeh et al., 2017), focusing on the ROC story cloze task.”
What Makes Reading Comprehension Questions Easier? (2018, background): “Our motivation originates from studies that demonstrated unintended biases in the sourcing of other NLU tasks, in which questions contain simple patterns and systems can recognize these patterns to answer them (Gururangan et al., 2018; Mostafazadeh et al., 2017).”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, background): “Despite their simplicity, these stories pose a signiﬁcant challenge to natural language understanding models (Mostafazadeh et al., 2017).”
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (2020, background): “It also lends itself naturally to commonsense reasoning tasks, where most datasets are based on crowdsourced narratives (MCScript (Ostermann et al. 2018), RocStories (Mostafazadeh et al. 2017)).”
Language Models are General-Purpose Interfaces (2022, methodology): “We conduct experiments of in-context learning on four categories: • Cloze and completion tasks: StoryCloze (Mostafazadeh et al., 2017), HellaSwag (Zellers et al., 2019) • Winograd-style tasks: Winograd (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020) • Commonsense reasoning:…”
A Multi-Attention based Neural Network with External Knowledge for Story Ending Predicting Task (2018, result): “We compare our model with some distinctive methods and approaches that rank high in LSDSem 2017 Shared Task (Mostafazadeh et al., 2017) in Tabel 1.”
Discovering Latent Knowledge in Language Models Without Supervision (2022, methodology): “We test models on 10 datasets: sentiment classification (IMDB (Maas et al., 2011) and Amazon (McAuley & Leskovec, 2013)), topic classification (AG-News (Zhang et al., 2015) and DBpedia-14 (Lehmann et al., 2015)), NLI (RTE (Wang et al., 2018) and QNLI (Rajpurkar et al., 2016)), story completion (COPA”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Yet, for additional interpretability, we also provide some ZeroShot accuracy results for LAMBADA [35], ARC (Easy and Challenge) [2], PIQA [44] and StoryCloze [33].”
Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “, 2020], StoryCloze [Mostafazadeh et al., 2017]), Classification (DBPedia [Zhang et al.”
Scaling Data-Constrained Language Models (2023, background): “co/datasets/sciq StoryCloze 2016 [67] test 1871 0.”
DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “Following the previous work [5], we choose Winogrande [49], Winograd [50], Storycloze [51], and Hellaswag [52] as the benchmark.”
Event Representations for Automated Story Generation with Deep Neural Nets (2017, methodology): “Speciﬁcally, we use an n -gram length of 4 and so the score takes into account all n -gram overlaps between the generated and expected output where n varies from 1 to 4 (Papineni et al. 2002).”