Tackling the Story Ending Biases in The Story Cloze Test

By Rishi Sharma, James F. Allen, Omid Bakhshandeh, Nasrin Mostafazadeh. 2018

87 citations recorded by the source corpus.

The Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. There have been a variety of models tackling the SCT so far. Although the original goal behind the SCT was to require systems to perform deep language understanding and commonsense reasoning for successful narrative understanding, some recent models could perform significantly better than the initial baselines by leveraging human-authorship biases discovered in the SCT dataset. In order to shed some light on this issue, we have performed various data analysis and analyzed a variety of top performing models presented for this task. Given the statistics we have aggregated, we have designed a new crowdsourcing scheme that creates a new SCT dataset, which overcomes some of the biases. We benchmark a few models on the new dataset and show that the top-performing model on the original SCT dataset fails to keep up its performance. Our findings further signify the importance of benchmarking NLP systems on various evolving test sets.

Subfields: Developmental and Educational Psychology, Education, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Developmental and Educational Psychology

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “…causes and endings given a sentence (COPA; Roemmele et al., 2011), for choosing the most sensible ending to a short story (RocStories; Mostafazadeh et al., 2016; Sharma et al., 2018), and for predicting likelihood of a hypothesis by regressing to an ordinal label (JOCI; (Zhang et al., 2017)).”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[1] chose a set of tasks (and corresponding datasets) to hold out from the multitask training mixture – specifically, sentence completion (COPA [37], H-SWAG [38], and Story Cloze [39] datasets), natural language inference (ANLI [40], CB [41], and RTE [42]), coreference resolution (WSC [43] and Winog”
Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation (2019, methodology): “The Story Cloze Test and ROCStories Corpora (Mostafazadeh et al., 2016; Sharma et al., 2018) require to ﬁgure out the right ending from two candidate sentences after a four-sentence story.”
TIES-Merging: Resolving Interference When Merging Models (2023, methodology): “…base model and finetune (IA) 3 models on the train split of eleven datasets including sentence completion (COPA [61], H-SWAG [88], and Story Cloze [68] datasets), natural language inference (ANLI [49], CB [44], and RTE [11]), coreference resolution (WSC [37] and Winogrande [64]), and word sense…”
GLUCOSE: GeneraLized and COntextualized Story Explanations (2020, background): “ently narrow and prone to hidden curation biases (Sharma et al., 2018; Belinkov et al., 2019).”
SemEval-2020 Task 4: Commonsense Validation and Explanation (2020, methodology): “The Story Cloze Test and ROCStories Corpora (Mostafazadeh et al., 2016; Sharma et al., 2018) aim to ﬁgure out the right ending from two candidate sentences after a four-sentence story.”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “(Mostafazadeh et al., 2016; Sharma et al., 2018) is a 2-way classiﬁcation task designed to test commonsense reasoning.”