STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation

By Nader Akoury, Shufan Wang, Joshua Whiting, Stephen Hood, Nanyun Peng, Mohit Iyyer. 2020

98 citations recorded by the source corpus.

Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semi-structured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.

Subfields: Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “tasks and retrieval-based models, Storium (Akoury et al., 2020) which focuses on open-ended story generation, and BIG bench3 which focuses on measuring few-shot and zero-shot capabilities of language models.”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “Other evaluation methods ask the evaluators to directly interact with the generated text; for example, Choose Your Own Adventure (Clark and Smith, 2021) and Storium (Akoury et al., 2020) evaluate story generation models by having people write stories with the help of generated text.”
Hurdles to Progress in Long-form Question Answering (2021, background): “Crowd-workers are unlikely to spend time reading & analyzing long text (Akoury et al., 2020).”
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (2021, methodology): “The positive reactions from participants indicate this format could work well on alternative crowdsourcing platforms, like LabintheWild,10 or launched as an independent writing game, similar to Akoury et al. (2020).”
Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, background): “, 2018) resulting in only a fraction of generated text being retained (Akoury et al., 2020).”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “…and find that on average about 42% of our ratings are filtered out (ranging from 20%-72% across all experiments).11 Of our surveyed papers, only Akoury et al. (2020) report actual work time, demonstrating that this is a major issue in modern AMT evaluations of text quality that most…”
Cut the CARP: Fishing for zero-shot story evaluation (2021, unknown): “Recently, researchers interested in story evaluation have developed more sophisticated techniques for evaluating stories (Akoury et al., 2020; Dou et al., 2021) as well as the Purdy Index (Purdy et al., 2018) and the Fabula-Entropy Index (Castricato et al., 2021a;b).”
Gender and Representation Bias in GPT-3 Generated Stories (2021, background): “Advances in large language models have allowed new possibilities for their use in storytelling, such as machine-in-the-loop creative writing (Clark et al., 2018; Kreminski et al., 2020; Akoury et al., 2020) and narrative generation for games (Raley and Hua, 2020).”