DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts

By Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, Yejin Choi. 2021

533 citations recorded by the source corpus.

Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DExperts: Decoding-time Experts, a decoding-time method for controlled text generation that combines a pretrained language model with “expert” LMs and/or “anti-expert” LMs in a product of experts. Intuitively, under the ensemble, tokens only get high probability if they are considered likely by the experts, and unlikely by the anti-experts. We apply DExperts to language detoxification and sentiment-controlled generation, where we outperform existing controllable generation methods on both automatic and human evaluations. Moreover, because DExperts operates only on the output of the pretrained LM, it is effective with (anti-)experts of smaller size, including when operating on GPT-3. Our work highlights the promise of tuning small LMs on text with (un)desirable attributes for efficient decoding-time steering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Given recent development of inference-time meth-ods for control that can reduce toxicity (e.g., PPLM (Dathathri et al., 2019), GeDi (Krause et al., 2020), DExperts (Liu et al., 2021)), there is potential for extending these methods to bias mitigation.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “DExperts (Liu et al., 2021) achieve detoxification by adopting two generative models (expert and anti-expert models) to replace the original discriminator, inspired by Bayes theorem and GeDi (Krause et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “…the probabilities of harmful text detectors directly, composing them with LM output probabilities to up-weight or down-weight the probabilities of each token based on generated left context (Yang and Klein, 2021; Liu et al., 2021a; Dathathri et al., 2019; Krause et al., 2020; Schick et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Many strategies have been developed to address some of these harms (e.g., [58, 4, 48, 36, 34, 19, 60]).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “However, designing prompts is notoriously challenging (Liu et al., 2021c).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “, 2021), while others explicitly change the token distribution during generation (Dathathri et al., 2019; Krause et al., 2021; Liu et al., 2021).”
Using In-Context Learning to Improve Dialogue Safety (2023, background): “Finally, Liu et al. (2021) proposed a product-of-experts-based procedure for detoxified generation.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “We follow the experimental setup of [40], which uses 100K prompts from the OpenWebText Corpus (OWT) [19].”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, methodology): “Other inference-time methods (Dathathri et al., 2020; Liu et al., 2021) may be used to steer text generation with DEMIX experts.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Liu et al. (2021) detoxify a language model’s output by upweighting the probabilities of generating words considered unlikely by a second “anti-expert” model that models toxic language.”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, methodology): “, 2022) feedback, anti-experts (Liu et al., 2021) and baked-in safety models (Xu et al.”
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, methodology): “To improve dialogue safety, numerous work researches on generation detoxifying (Dinan et al., 2019; Smith et al., 2020a; Liu et al., 2021a).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “Current state-of-the-art models typically fine-tune entire pre-trained LMs (e.g., Ziegler et al., 2019a; Keskar et al., 2019; Ziegler et al., 2019b; Liu et al., 2021e).”
Systematic Rectification of Language Models via Dead-end Analysis (2023, methodology): “Similarly, Liu et al. (2021) uses a toxic LM as an “anti-expert” and a non-toxic LM as an “expert” to boost the probability of non-toxic tokens.”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “RLHF), and the state-of-the-art controlled generation approaches GeDi [17] and DEXPERTS [21].”
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models (2022, unknown): “DE XPERT [14] controls the generation of large-scale pre-trained LM with an “expert” LM trained on non-toxic data and “anti-expert” LM trained on toxic data in a product of experts [30].”
Prompt Consistency for Zero-Shot Task Generalization (2022, methodology): “For the training-time tuning scenario, we use up to 10,000 data points from the training set for training if the train set contains more than 10,000 data points.”