Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task

By Zheng Jon Cai, Lifu Tu, Kevin Gimpel. 2017

69 citations recorded by the source corpus.

We consider the ROC story cloze task (Mostafazadeh et al., 2016) and present several findings. We develop a model that uses hierarchical recurrent networks with attention to encode the sentences in the story and score candidate endings. By discarding the large training set and only training on the validation set, we achieve an accuracy of 74.7%. Even when we discard the story plots (sentences before the ending) and only train to choose the better of two endings, we can still reach 72.5%. We then analyze this “ending-only” task setting. We estimate human accuracy to be 78% and find several types of clues that lead to this high accuracy, including those related to sentiment, negation, and general ending likelihood regardless of the story context.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “These stylistic features are an obvious case of annotation artifacts (Cai et al., 2017; Schwartz et al., 2017).”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “Annota-tors can accidentally leave unintended shortcuts in datasets that allow models to achieve high accuracy by effectively “cheating” (Cai et al., 2017; Gururangan et al., 2018; Min et al., 2019).”
Story Ending Prediction by Transferable BERT (2019, methodology): “Neural models [Cai et al. , 2017; Zhou et al. , 2019] for SCT learn embeddings for the story context and candidate endings, and select the right ending by computing the embeddings’ similarity.”
Annotation Artifacts in Natural Language Inference Data (2018, background): “Schwartz et al. (2017) and Cai et al. (2017) showed that certain biases are prevalent in the ROC stories cloze task (Mostafazadeh et al., 2016), which allow models trained on the endings alone, and not the story preﬁx, to yield state-of-the-art results.”
Are NLP Models really able to Solve Simple Math Word Problems? (2021, background): “Cai et al. (2017) showed that biases prevalent in the ROC stories cloze task allowed models to yield state-of-the-art results when trained only on the endings.”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “Annotation artifacts in the Story Cloze Test, for example, allow models to achieve high performance while only looking at the proposed endings and ignoring the stories (Schwartz et al., 2017; Cai et al., 2017).”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “It has also been shown that features focusing on sentence length, sentiment, and negation are sufﬁcient for achieving high accuracy on this dataset (Schwartz et al., 2017a; Cai et al., 2017; Bugert et al., 2017).”
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (2019, background): “Many recent studies have suggested the importance of measuring the dataset bias by checking the model performance based on partial information of the problem (Gururangan et al., 2018; Cai et al., 2017).”
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task (2017, neutral): “8One exception is the EndingsOnly system (Cai et al., 2017), which was published in concurrence with this work, and obtains roughly the same results.”
Tackling the Story Ending Biases in The Story Cloze Test (2018, unknown): “, (Cai et al., 2017)”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “These cues are even noticeable by humans (Cai et al., 2017).”
A Multi-Attention based Neural Network with External Knowledge for Story Ending Predicting Task (2018, methodology): “The common strategies utilized to perform the task can be classiﬁed into two kinds of approaches: (1) traditional machine learning techniques with optimal feature engineering; and (2) deep learning-based models with effective strategies (Cai et al., 2017).”
There is Strength in Numbers: Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020, background): “Similarly, in the ROCSto-ries corpus (Mostafazadeh et al., 2016), Schwartz et al. (2017); Cai et al. (2017) show it is possible to achieve non-trivial prediction accuracy by only considering candidate endings and without taking the stories in account.”
SentiLARE: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis (2019, background): “Generalization to Other NLP Tasks : Since sentiment is a common feature to improve the tasks of text matching (Cai et al., 2017; Li et al., 2019), we chose three text matching tasks to explore whether our sentiment-aware representations can also ben-eﬁt these tasks, including story ending…”
Story Ending Selection by Finding Hints From Pairwise Candidate Endings (2019, unknown): “the original Story Cloze Test dataset, similar to recent works [8], [9], [22].”