CTRL: A Conditional Transformer Language Model for Controllable Generation
By Nitish Shirish Keskar, Bryan McCann, L. Varshney, Caiming Xiong, Richard Socher. 2019
1,445 citations recorded by the source corpus.
Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model-based source attribution. We have released multiple full-sized, pretrained versions of CTRL at this https URL.
Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Graham Neubig
- Noah A. Smith
- Yoshua Bengio
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Noah A. Smith
- Timo Schick
- Hannaneh Hajishirzi
- Michael Lewis
- Jianfeng Gao
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Yejin Choi
- Hannaneh Hajishirzi
- Hinrich Schütze
- Timo Schick
- Michael Lewis
- Noah A. Smith
- Daniel Klein
Citations in the corpus
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…are various approaches that augment LMs with some form of additional textual information during pretraining, including various forms of metadata (Keskar et al., 2019), HTML tags (Aghajanyan et al., 2021), Wikipedia markup (Schick et al., 2022), or related texts obtained from an information…”
- Experience Grounds Language (2020, background): “…galvanized progress in NLP. Performance peaks have been reached in tasks such as language modeling (Radford et al., 2019; Zellers et al., 2019c; Keskar et al., 2019) and span-selection question answering (Devlin et al., 2019a; Yang et al., 2019; Lan et al., 2019) through massive data and…”
- Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “, 2020) or conditional LMs (Keskar et al., 2019) will likely exhibit the same type of biases.”
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “ATCON Following the training approach used for CTRL (Keskar et al., 2019), we prepend the appropriate attribute token to each example in our randomly-sampled corpus.”
- Evaluating Large Language Models Trained on Code (2021, background): “Another possibility is to label the pre-training data based on code quality, then condition the model on the ’high quality’ label at deployment time (Keskar et al., 2019).”
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021, background): “Recent work in the literature has focused on controlling natural language generation using traditional approaches from machine learning like novel architectures which condition outputs [15, 16], more advanced sampling techniques [6, 11], gradient-based”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
- The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, methodology): “The repeating problem has been observed in nearly all text generation models [Keskar et al., 2019, Shumailov et al., 2021] and to rule this out as the cause of Model Collapse , we further provide numerical experiments when models are explicitly encouraged to produce non-repeating sequences with…”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “As an efficient method, CTRL (Keskar et al., 2019) pre-pended con-trol codes before sentences in training corpora, which is a direct and effective method to model P θ ( x t | x t , c ) , where c is the desired attribute that is formalized into control codes.”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Along similar lines, Keskar et al. (2019); Gururangan et al. (2020); Chan et al. (2021) adopt a setup of training class-conditional LMs where each LM accepts "control codes" as an additional input.”
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “For example, Keskar et al. (2019) and Dathathri et al. (2019) both look at training large-scale controllable language models.”
- BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, methodology): “In experiments, we evaluate biases in open-ended English language generation with three common LMs: GPT-2 [27], BERT [10], and CTRL with the Wikipedia (CTRL-WIKI), thoughts (CTRL-THT), and opinion (CTRL-OPN) control codes [15].”
- Recipes for Safety in Open-domain Chatbots (2020, background): “Keskar et al. (2019) train a large-scale controllable model that can modulate generations through con-trol tokens, but also don’t look at offensiveness.”
- How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Keskar et al. (2019) trained CTRL, a large (1.63B parameters) pretrained language model that can be conditioned on style or content for controlling generated text.”
- Few-Shot Text Generation with Natural Language Instructions (2020, background): “Keskar et al. (2019) and He et al. (2020) similarly use prompts and keywords for controllable text generation, but require specific pretraining procedures and do so only in high-resource settings.”