A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
By Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, Minlie Huang. 2020
268 citations recorded by the source corpus.
Story generation, namely, generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we use multi-task learning, which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Generation, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Noah A. Smith
- Zhiyuan Liu
- Jason Aaron Edward Weston
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jianfeng Gao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Alan L. Yuille
- Jitendra Malik
In Natural Language Generation
- Minlie Huang
- Marzena Karpinska
- Nader Akoury
- Xiang Ren
- Yejin Choi
- Mohit Iyyer
- Furu Wei
- Bill Yuchen Lin
In Knowledge Graphs & Representation
- Xiang Ren
- Minlie Huang
- Bill Yuchen Lin
- Chenguang Zhu
- Meng Jiang
- Jian Guan
- Zhiyuan Liu
- Tim Rocktaschel
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “…re-writing, among others) (Raffel et al., 2020; Zhang et al., 2020b; Aghajanyan et al., 2022; Prabhumoye et al., 2018), (2) or open-ended tasks (such as dialogue generation, prompt-based autocompletion, story generation, and more) (Adiwardana et al., 2020; Brown et al., 2020; Guan et al., 2020).”
- Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, methodology): “Specifically, we focus on automatic knowledge graph construction and story generation and quantify biases in COMeT (Bosselut et al., 2019) and CommonsenseStoryGen (CSG) (Guan et al., 2020).”
- Pre-Trained Models: Past, Present and Future (2021, background): “…and relations to raw text is often troublesome and can introduce noise in data pre-processing, another line of works (Bosselut et al., 2019; Guan et al., 2020; Chen et al., 2020e) can directly convert structural knowledge into the serialized text and let models learn knowledge-text…”
- Pre-trained models for natural language processing: A survey (2020, methodology): “[51] employed commonsense knowledge bases, ConceptNet and ATOMIC, to enhance GPT-2 for story generation.”
- UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, methodology): “We analyzed 381 unreasonable stories generated by various NLG models like Plan&Write (Yao et al., 2019) and fine-tuned GPT-2 (Radford et al., 2019) base on ROCStories (Mostafazadeh et al., 2016), and summarized four major types of errors, including repeated plots (repeating similar texts), poor coher”
- Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision (2020, methodology): “To further instill common-sense knowledge into the LM, we follow Guan et al. (2020) and continue pre-training GPT2-M on triplets from Con-ceptNet (Speer, Chin, and Havasi 2017) converted to nat-ural language using the templates from Davison, Feldman, and Rush (2019).”
- Content Planning for Neural Story Generation with Aristotelian Rescoring (2020, background): “Guan et al. (2020); Mao et al. (2019) use a similar approach, however they incorporate structured commonsense knowledge from external datasets or knowledge bases to improve a story generated from a prompt.”
- STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation (2020, methodology): “…to generate stories by training neural models with limited context (Peng et al., 2018; Fan et al., 2018; Goldfarb-Tarrant et al., 2019) and structured knowledge, either external (Mao et al., 2019; Guan et al., 2020; Goldfarb-Tarrant et al., 2020) or derived (Yao et al., 2019; Fan et al., 2019).”
- The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, methodology): “Model-generated stories: We follow a similar modeling approach to prior story generation work (Mao et al., 2019; Guan et al., 2020) by fine-tuning a pretrained GPT-2 medium-sized model (Radford et al.”
- How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “Works such as (Bosselut et al., 2019; Guan et al., 2020) focus on pre-training transformer-based language learning systems with large-scale commonsense knowledge graphs such as ATOMIC (Sap et al., 2019) and ConceptNet (Speer and Havasi, 2012) for use in knowledge graph completion and story ending…”
- Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “…pre-computed knowledge embeddings into language generation (Zhou et al., 2018); transferring knowledge into language model with triplet information (Guan et al., 2020a); performing reasoning over knowledge graph via path finding strategies (Liu et al., 2019c; Ji et al., 2020a; Yu et al., 2022d);…”
- Automated Storytelling via Causal, Commonsense Plot Ordering (2020, background): “Guan, Wang, and Huang (2019) incorporate external commonsense knowledge sources to explicitly improve story ending generation and Mao et al. (2019); Guan et al. (2020) look at fine tuning pre-trained transformer based language models (Vaswani et al. 2017) on commonsense sources like ConceptNet…”
- A Survey of Knowledge-enhanced Text Generation (2020, background): “ACM Reference Format: Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang.”
- Language Generation with Multi-hop Reasoning on Commonsense Knowledge Graph (2020, methodology): “In language generation, previous work (Bhagavatula et al., 2020; Guan et al., 2020) transfers common-sense knowledge into pre-trained language models by utilizing triple information in commonsense knowledge bases such as ConceptNet (Speer and Havasi, 2012) and ATOMIC (Sap et al., 2019).”
- Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “To enrich the content of generated text, some work aimed to incorporate external knowledge into PLMs. Guan et al. [79] and Mao et al. [161] utilized a commonsense knowledge base to fine-tune PLMs to generate reasonable stories.”
- Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, background): “4-gram repetition (Guan et al., 2020), measures the fraction of stories with at least one repeated 4-gram”
- Explaining Toxic Text via Knowledge Enhanced Text Generation (2022, methodology): “Zhang et al. (2019) use knowledge graph embeddings to model conversation flow, while Guan et al. (2020) use triples extracted from knowledge graphs to enhance story generation.”
- Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, unknown): “, 2020) and the fine-tuning stage (Yang et al., 2019; Guan et al., 2020).”
- Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022, methodology): “For example, Guan et al. (2020) conducted post-training on sy-thetic data constructed from commonsense KG by translating triplets into natural language texts using templates.”
- Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, background): “[46] Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang.”