Hierarchical Neural Story Generation

By Angela Fan, Michael Lewis, Yann Dauphin. 2018

2,050 citations recorded by the source corpus.

We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “In preliminary experiments, we observed that for some tasks such as story generation [11], the retrieval component would “collapse” and learn to retrieve the same documents regardless of the input.”
Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “Top-k restricts the distribution to all but the k most likely tokens, where k is a constant (Fan et al., 2018).”
GLTR: Statistical Detection and Visualization of Generated Text (2019, background): “, 2017), k-max sampling (Fan et al., 2018), beam search (Chorowski and Jaitly, 2016; Shao et al.”
Defending Against Neural Fake News (2019, background): “We also compare with top-k sampling, wherein the most probable k tokens are used at each timestep (Fan et al., 2018).”
The Curious Case of Neural Text Degeneration (2019, methodology): “In this study we use the GPT model (Radford et al., 2018).2 Dataset For open-ended generation, we use the WritingPrompts dataset of (Fan et al., 2018).”
CoQA: A Conversational Question Answering Challenge (2018, background): “…Project Gutenberg,5 middle and high school English exams from RACE (Lai et al., 2017), news articles from CNN (Hermann et al., 2015), articles from Wikipedia, Reddit articles from the Writing Prompts dataset (Fan et al., 2018), and science articles from AI2 Science Questions (Welbl et al., 2017).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Recent work in sampling methods for text generation has focused on reducing repetition by replacing it with novel, coherent text (Fan et al., 2018; Holtzman et al., 2019).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “…2017; Baevski and Auli, 2019), abstractive document summarization (Fan et al., 2018a; Liu et al., 2018; Narayan et al., 2018), story generation (Fan et al., 2018b, 2019), error correction (Chollampatt and Ng, 2018), multilingual sentence embeddings (Artetxe and Schwenk, 2018), and dialogue…”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “We use the hyperparameters from the original papers for each technique, except we generate using topk sampling (Fan et al., 2018) with k = 50 for all methods to enable a fair comparison.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “In practice, various strategies exist for sampling tokens, including random sampling, sampling from the top-k highest probability tokens (Fan et al., 2018), or sampling after using a temperature to sharpen the next-token distribution.”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “• Top-k sampling (Fan et al., 2018): at each time step, re-distribute the probability mass of the top k words with highest probabilities and sample.”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021, background): “Recent work in the literature has focused on controlling natural language generation using traditional approaches from machine learning like novel architectures which condition outputs [15, 16], more advanced sampling techniques [6, 11], gradient-based”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “Writing Prompts (Fan et al., 2018) 34.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Recently, a convolutional alternative to sequence modeling has been proposed showing promise for machine translation (Gehring et al., 2017a,b) and story generation (Fan et al., 2018).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “introduced recently to improve the quality of LMgenerated text (Graves, 2012; Fan et al., 2018; Holtzman et al., 2020; Meister et al., 2022).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “Without ALICE, we use top-k decoding (Fan et al., 2018) alone with our toxic and benign prompts.”