Do Massively Pretrained Language Models Make Better Storytellers?

By Abigail See, Aneesh S. Pappu, Aneesh S. Pappu, Rohun Saxena, Akhila Yerukola, Christopher D. Manning. 2019

185 citations recorded by the source corpus.

Large neural language models trained on massive amounts of text have emerged as a formidable strategy for Natural Language Understanding tasks. However, the strength of these models as Natural Language Generators is less clear. Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018). By evaluating the generated text across a wide variety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers. We find that although GPT2-117 conditions more strongly on context, is more sensitive to ordering of events, and uses more unusual words, it is just as likely to produce repetitive and under-diverse text when using likelihood-maximizing decoding algorithms.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “In fact, See et al. (2019) note that it takes setting k to 1000 to achieve about the same amount of rare word usage and fraction of non-stopword text as as human writing.3 This makes it very easy for the model to pick out machine-generated text based on these distributional differences.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “For specific tasks, evaluating quality of generated output compared to human output can give signal to the likelihood of a generative AI system replacing human labor [212].”
Enabling Language Models to Fill in the Blanks (2020, background): “…text input with blanks, predicts the answers, and then combines them to produce the output. ing remarkably coherent text (Zellers et al., 2019; See et al., 2019), (2) efﬁcient at generating text, and (3) conceptually simple, but cannot inﬁll effectively as they can only leverage context in a…”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “Modern LMs suffer from the “likelihood trap” (See et al., 2019; Zhang et al., 2021), in which high likelihood (low perplexity) sequences produced by greedy decoding or beam search tend to be dull and repetitive.”
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020, methodology): “…(underlined , e.g., ﬁrst stopped driving and then drove to the party ), in spite of their good ﬂuency and intra-sentence coherence. with generic plots (See et al., 2019) (e.g., I have a great time ), even when using the powerful generative model OpenAI’s GPT-2 (Radford et al., 2019), as shown in…”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, methodology): “model-generated text, which is critical for openended tasks given the unreliability of automatic metrics (Peng et al., 2017; Reiter, 2018; See et al., 2019), is frequently conducted on Amazon’s popular Mechanical Turk platform (AMT) to minimize cost and time.”
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models (2021, background): “Some connectives may have the same sense depending on the specific context in which they appear (Stede, 2012; Webber et al., 2003), for instance both since and while may bear a temporal interpretation.”
Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, background): “This has spurred increasing interest in computer-assisted creativity (See et al., 2019; Elkins and Chun, 2020; Ramesh et al., 2022; Branwen, 2020) such as building co-creative assistants for writing stories, poems, and argumentative essays (Lee et al., 2022; Swan-son et al., 2021; Uthus et al.,…”
Human and Automatic Detection of Generated Text (2019, background): “In fact, See et al. (2019) note that it takes setting k to 1000 to achieve about the same amount of rare word usage and fraction of non-stopword text as as human writing 2 This makes it very easy for the model to pick out machine-generated text based on these distributional differences.”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “They strongly condition on the preceding tokens, produce coherent and contentful text (See et al., 2019), and importantly, capture some commonsense and world knowledge (Davison et al.”
Typical Decoding for Natural Language Generation (2022, background): “For many language generation tasks, decoding strategies which aim to ﬁnd maximal probability strings produce text that is undesirable—e.g., generic or degenerate (Holtzman et al., 2020; See et al., 2019; Eikema and Aziz, 2020; Zhang et al., 2021; DeLucia et al., 2021).”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, background): “They have also shown great success when fine-tuned to particular text generation tasks (See et al., 2019; Zhang et al., 2019; Keskar et al., 2019).”
Relating Neural Text Degeneration to Exposure Bias (2021, background): “), a pre-trained language model that has been shown useful in many NLP tasks (Radford, 2018; Zhang et al., 2019b; Petroni et al., 2019; Talmor et al., 2019; See et al., 2019).”
Collaborative Storytelling with Large-scale Neural Language Models (2020, background): “[See et al. 2019] conduct an in-depth analysis of the storytelling capabilities of large-scale neural language models.”
Cut the CARP: Fishing for zero-shot story evaluation (2021, background): “Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story (See et al., 2019; Xu et al., 2020; Nichols et al., 2020; Fang et al., 2021; Hazarika et al., 2021).”
Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, methodology): “We choose k = 1000 to produce better story results with less repetition in transformer models as stated in (See et al., 2019).”