The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

By Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, Noah A. Smith. 2017

121 citations recorded by the source corpus.

A writer’s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “In light of recent work showing that crowd-sourced data often contains artifacts which can be exploited to perform well without solving the intended task (Schwartz et al., 2017; Gururan-gan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018), we audit the data for such artifacts.”
Annotation Artifacts in Natural Language Inference Data (2018, background): “Schwartz et al. (2017) and Cai et al. (2017) showed that certain biases are prevalent in the ROC stories cloze task (Mostafazadeh et al., 2016), which allow models trained on the endings alone, and not the story preﬁx, to yield state-of-the-art results.”
Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, methodology): “Few-shot Learning with Language Models Recent work uses LMs to solve NLP tasks, e.g., for story cloze prediction (Schwartz et al., 2017), knowledge base comple- tion (Petroni et al., 2019), and Winograd schemas (Trinh & Le, 2018).”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “However, crowdsourcing as an unconstrained annotation approach is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out-of-distribution generalisation (Chen et al., 2016; Weissenborn…”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “Data Generation While manual dataset creation is expensive and complex (Schwartz et al., 2017; Agrawal et al., 2018; Tsuchiya, 2018; Bras et al., 2020), crowdsourcing is the most popular method for goal-oriented, high quality/coverage datasets.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “, 2019), especially those that minimize annotation artifacts and spurious correlations (Schwartz et al., 2017; Gururangan et al., 2018; Le et al., 2019).”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “It has also been shown that features focusing on sentence length, sentiment, and negation are sufﬁcient for achieving high accuracy on this dataset (Schwartz et al., 2017a; Cai et al., 2017; Bugert et al., 2017).”
Are We Done with MMLU? (2024, neutral): “Benchmark issues can arise from biases in the framing of the task [26]; noisy annotations [27]; web-crawls [28, 29, 30]; automated labelling processes such as crowdsourcing [31] — where annotators may be optimizing for their own incentives; human errors [32] — e.g., lack of expertise on a given…”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Moreover, it has the potential of introducing annotation artifacts: subtle patterns that are by themselves highly predictive of the ‘correct’ or ‘incorrect’ label [72, 28, 61].”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “ates are discarded, resulting in a loss of ˇ1% of the data. 3.2 Mitigating Dataset Biases Dataset creation is always fraught with the risk of inducing unintended errors and biases (Chen et al., 2016; Schwartz et al., 2017). As Hewlett et al. (2016) only carried out limited analysis of their WIKIREAD”
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (2019, neutral): “In this paper, we continue recent efforts to understand biases that are introduced during the process of data creation (Levy et al., 2015; Schwartz et al., 2017; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Aharoni and Goldberg, 2018; Paun et al., 2018).”
Commonsense Knowledge Mining from Pretrained Models (2019, methodology): “Schwartz et al. (2017) and Trinh and Le (2018) demonstrates a similar approach to using language models for tasks requiring commonsense, such as the Story Cloze Task and the Winograd Schema Challenge, respectively (Mostafazadeh et al., 2016; Levesque et al., 2012).”
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task (2017, methodology): “We achieve 72.5%, matching the similar ending-only result of Schwartz et al. (2017b).”
Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation (2019, methodology): “This worry was raised from the fact that models can achieve high performance in the ROC Story Cloze Task when only looking at the alternative endings ignoring the stories (Schwartz et al., 2017).”
Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer (2018, background): “…many prior works on linguistic style analysis conﬁrm our observation that attributes often manifest in idiosyncratic phrases (Recasens et al., 2013; Schwartz et al., 2017; Newman et al., 2003), we recognize the fact that in some problems (e.g., Pavlick and Tetreault (2017)), content and attribute…”
Story Cloze Task: UW NLP System (2017, background): “The reader is referred to Schwartz et al. (2017) for more details and discussion.”
Challenges in Automated Debiasing for Toxic Language Detection (2021, background): “Second, the dataset biases in NLU are predominantly artifacts introduced during data creation (e.g., negations, exaggerations; Schwartz et al., 2017; Gururangan et al., 2018), whereas those in toxic language detection are grounded in the social dynamics of the world (Spears, 1998; Tech-nau, 2018).”