Controllable Neural Story Plot Generation via Reward Shaping

By Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, Mark O. Riedl. 2018

99 citations recorded by the source corpus.

Language-modeling--based approaches to story plot generation attempt to construct a plot by sampling from a language model (LM) to predict the next character, word, or sentence to add to the story. LM techniques lack the ability to receive guidance from the user to achieve a specific goal, resulting in stories that don't have a clear sense of progression and lack coherence. We present a reward-shaping technique that analyzes a story corpus and produces intermediate rewards that are backpropagated into a pre-trained LM in order to guide the model toward a given goal. Automated evaluations show our technique can create a model that generates story plots which consistently achieve a specified goal. Human-subject studies show that the generated stories have more plausible event ordering than baseline plot generation techniques.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Some others use heuristic rewards as in (Li et al., 2016b; Tambwekar et al., 2019), in order to improve certain a priori desirable features of generated stories or dialogues.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Although several studies have considered hand-crafting general rewards to ensure desirable features like fluency (Liu et al., 2016a; Tambwekar et al., 2019), coming up with complete or perfect rewards is highly nontrivial (Wu et al., 2016; Vedantam et al., 2015).”
A Distributional Approach to Controlled Text Generation (2020, methodology): “, 2017), or hand crafted rewards (Li et al., 2016b; Tambwekar et al., 2019) to improve certain a priori desirable features.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “In the domain of open-text generation, REINFORCE [74] and PPO [2] have been used for controllable story generation, and soft Q-Learning [20] has been applied to generate prompts for steering language model generations.”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “…in multiple language generation tasks ranging from translation (Choshen et al., 2019), summarisation (Paulus et al., 2018), story generation (Tambwekar et al., 2019), video captioning (Pa-sunuru and Bansal, 2017), dialogue (Jaques et al., 2019), to code generation (Korbak et al., 2021) and…”
Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021, unknown): “…2017; Fu et al., 2018; Peng et al., 2018; Dai et al., 2019), tense (Logeswaran et al., 2018), plots (Ammanabrolu et al., 2020; Fan et al., 2019; Tambwekar et al., 2019; Yao et al., 2019; Goldfarb-Tarrant et al., 2019, 2020), societal bias (Wallace et al., 2019; Sheng et al., 2020b,a), and…”
Automated Storytelling via Causal, Commonsense Plot Ordering (2020, methodology): “The questions we use are adapted from Purdy et al. (2018) and have been used in multiple storytelling works as an indication of story quality (Tambwekar et al. 2019; Ammanabrolu et al. 2020b).”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “Storystructure[32,33,40],storyending[85,132], topic[16,73,141,153],persona[15,76] DatatoText Input: [78],genderbias[28,104],subjectivebias[101], socialbias[4,121],sentimentbias[51],toxicity[62,74]) FormatControl Input:DesiredFormat,prompttext Output:Textinpre-definedformation Format[66,70,120,122]…”
Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, methodology): “Story generation can be controlled by conditioning generation on high-level plot outlines (Fan et al., 2018; Peng et al., 2018; Rashkin et al., 2020), story in-ﬁlling (Don-ahue et al., 2020; Wang et al., 2020), ﬁne-tuning on goals (Tambwekar et al., 2019a; Alabdulkarim et al., 2021).”
Text Editing by Command (2020, background): “, 2019), story ending (Tambwekar et al., 2019), and outlines (Rashkin et al.”
Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, methodology): “We focus on evaluating our system in the domain of science ﬁction plots (Ammanabrolu et al., 2020a), consistent with prior work (Tambwekar et al., 2019; Ammanabrolu et al., 2020a).”
Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “Others use heuristic rewards (for instance Li et al. (2016b) for dialogue generation and Tambwekar et al. (2019) for story generation) in order to obtain certain a priori desirable features of generated sequences that then incentivize good performance on target metrics.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Some studies used inference methods include variation inference, various modalities, ontology, emotion, and knowledge [114]–[121].”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “A number of prior works have explored combining online RL methods with language models for natural language tasks such as machine translation or summarization [19, 20, 21, 22, 23, 24, 25, 26, 27, 28].”