Controllable Story Generation with External Knowledge Using Large-Scale Language Models

By Peng Xu, M. Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro. 2020

140 citations recorded by the source corpus.

Existing pre-trained large language models have shown unparalleled generative capabilities. However, they are not controllable. In this paper, we propose MEGATRON-CNTRL, a novel framework that uses large-scale language models and adds control to text generation by incorporating an external knowledge base. Our framework consists of a keyword predictor, a knowledge retriever, a contextual knowledge ranker, and a conditional text generator. As we do not have access to ground-truth supervision for the knowledge ranker, we make use of weak supervision from sentence embedding. The empirical results show that our model generates more fluent, consistent, and coherent stories with less repetition and higher diversity compared to prior work on the ROC story dataset. We showcase the controllability of our model by replacing the keywords used to generate stories and re-running the generation process. Human evaluation results show that 77.5% of these stories are successfully controlled by the new keywords. Furthermore, by scaling our model from 124 million to 8.3 billion parameters we demonstrate that larger models improve both the quality of generation (from 74.5% to 93.0% for consistency) and controllability (from 77.5% to 91.5%).

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Large Language Models

In Natural Language Processing

Citations in the corpus

Adaptive Semiparametric Language Models (2021, methodology): “…that have been used for question answering (de Masson d’Autume et al., 2019; Guu et al., 2020; Xiong et al., 2021; Kassner and Schutze, 2020), controllable generation (Xu et al., 2020), machine translation (Bapna and Firat, 2019; Khandelwal et al., 2021), and oneshot learning (Kaiser et al., 2017).”
Cut the CARP: Fishing for zero-shot story evaluation (2021, background): “Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story (See et al., 2019; Xu et al., 2020; Nichols et al., 2020; Fang et al., 2021; Hazarika et al., 2021).”
Progressive Generation of Long Text with Pretrained Language Models (2021, background): “, 2019), storytelling (Fan et al., 2019; Yao et al., 2019; Xu et al., 2020), machine translation (Ford et al.”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “MEGATRON-CNTR [150] is a controllable story generation framework that combines external knowledge and PLM.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Megatron-Cntrl [192] used extracted keywords to retrieve knowledge sentences and then selected top-ranked sentences for story generation.”
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, background): “[122] Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro.”
A Plug-and-Play Method for Controlled Text Generation (2021, background): “Xu et al. (2020) propose a framework to exert hard control over language generation which, nevertheless, requires training three large transformer models.”
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (2021, methodology): “We compare FUSION and GPT2 to see how CYOA can evaluate two models with different underlying architectures; they are also both common story generation baselines (See et al., 2019; Xu et al., 2020; Rashkin et al., 2020).”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, methodology): “Each paper was published between 2018 and 2020 at ACL, NAACL, or EMNLP, and we exclude papers that use AMT to evaluate more wellestablished generation tasks like machine translation, or summarization.3 Unlike previous surveys of evaluating generated text (Çelikyilmaz et al., 2020; van der Lee et al.”