Plug and Play Language Models: A Simple Approach to Controlled Text Generation
By Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, Rosanne Liu. 2019
1,229 citations recorded by the source corpus.
Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Graham Neubig
- Hannaneh Hajishirzi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Ming-Hsuan Yang
- Mohit Bansal
- Xiaogang Wang
- Jitendra Malik
In Natural Language Processing
- Yejin Choi
- Mohit Bansal
- Luke S. Zettlemoyer
- Noah A. Smith
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Hannaneh Hajishirzi
In Prompting & Reasoning
- Yejin Choi
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Hinrich Schütze
- Timo Schick
- Michael Lewis
- Daniel Klein
- Percy Liang
Citations in the corpus
- Challenges in Detoxifying Language Models (2021, methodology): “Due to PPLM’s computational cost, and the need to filter out degenerate samples as in Dathathri et al. (2020) (cf. Appendix A.2), we only consider a subsample of 10K prompts for PPLM variants in the RTP automatic evaluation setting.”
- Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “, 2020) or incorporating a toxicity discriminator during decoding (Dathathri et al., 2020).”
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “Decoding with a Purpose Our analyses also highlight the promise of certain decoding methods, such as PPLM (Dathathri et al., 2020), which is among the most effective methods we tested at avoiding toxicity with toxic prompts.”
- Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Given recent development of inference-time meth-ods for control that can reduce toxicity (e.g., PPLM (Dathathri et al., 2019), GeDi (Krause et al., 2020), DExperts (Liu et al., 2021)), there is potential for extending these methods to bias mitigation.”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021;…”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, positive): “Rather than only token-level unsafety avoidance, PPLM (Dathathri et al., 2019) notices that P ( x | c ) ∝ P ( c | x ) , and adopts an attribute model to compute P ( c | x ) to guide the model decoding at the sentence level without any additional changes in the training phase.”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “1, such as toxicity/bias classifiers (Dathathri et al., 2019; Krause et al., 2021; Liu et al., 2021a), factuality metrics (Kryscinski et al.”
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “For example, Keskar et al. (2019) and Dathathri et al. (2019) both look at training large-scale controllable language models.”
- Recipes for Safety in Open-domain Chatbots (2020, methodology): “We note that a more sophisticated alternative is to generate responses chosen to not trigger a classifier, e.g. using the plug and play language model approach (Dathathri et al., 2019).”
- Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “In addition, work on building safer LLMs explores fine-tuning on curated data (Solaimon and Dennison, 2021) or directly controlling the generations of the model (Dathathri et al., 2019; Liu et al., 2021; Schick et al., 2021; Xu et al., 2020).”
- Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, background): “As a consequence, we can exert considerable control over the generated content by controlling the passage provided to the generator, potentially complementing controllable language generation using special tokens or attributes (Dathathri et al., 2020).”
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, neutral): “The control can also happen at decoding time, by weighted decoding (GeDi, Krause et al., 2020) or iteratively updating the past activations (PPLM, Dathathri et al., 2020).”
- Symbolic Music Generation with Diffusion Models (2021, background): “Conditional sampling from unconditional models: We build on top of the breadth of material that investigate steering generation in the latent space of an unconditionally trained generative model [37–40].”
- Towards Understanding and Mitigating Social Biases in Language Models (2021, background): “, (K (l) t−1, V (l) t−1)] where (K (i) t−1, V (i) t−1) corresponds to the key-value pairs from the i-th Transformer layer generated from time steps 0 to t − 1 (see (Dathathri et al., 2019) for more details).”
- GeDi: Generative Discriminator Guided Sequence Generation (2020, methodology): “Plug and Play LM (Dathathri et al., 2020, PPLM) applies up to 10 updates to the generating LM’s latent states per time step using gradients from a discriminator, also making it many times slower than generating from the LM directly.”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “The first family of approaches includes guided decoding methods (Dathathri et al., 2020; Qin et al., 2022), Monte Carlo sampling techniques such as rejection sampling to sample from simple distributions like pGDC bin (Roller et al., 2021; Ziegler et al., 2022), and Quasi Rejection Sampling (QRS)…”
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “These preferences could include topic and sentiment (Dathathri et al., 2020), valid musical notes and molecular structures (Jaques et al., 2017a), code compilability (Korbak et al., 2021), reducing distributional biases (Khalifa et al., 2021; Weidinger et al., 2021), evaluation metrics for Machine…”
- Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “, 2021), while others explicitly change the token distribution during generation (Dathathri et al., 2019; Krause et al., 2021; Liu et al., 2021).”
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “Adapting PLMs to generate texts of specific attributes can also be realized at inference time without any further training of the PLMs [10, 26, 27, 32, 47, 75].”