Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models

By Fatemehsadat Mireshghallah, Kartik Goyal, Taylor Berg-Kirkpatrick. 2022

91 citations recorded by the source corpus.

Recent work on controlled text generation has either required attribute-based fine-tuning of the base language model (LM), or has restricted the parameterization of the attribute discriminator to be compatible with the base autoregressive LM. In this work, we propose Mix and Match LM, a global score-based alternative for controllable text generation that combines arbitrary pre-trained black-box models for achieving the desired attributes in the generated text without involving any fine-tuning or structural assumptions about the black-box models. We interpret the task of controllable generation as drawing samples from an energy-based model whose energy values are a linear combination of scores from black-box models that are separately responsible for fluency, the control attribute, and faithfulness to any conditioning context. We use a Metropolis-Hastings sampling scheme to sample from this energy-based model using bidirectional context and global attribute features. We validate the effectiveness of our approach on various controlled generation and style-based text revision tasks by outperforming recently proposed methods that involve extra training, fine-tuning, or restrictive assumptions over the form of models.

Subfields: Artificial Intelligence, General Social Sciences, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Generative Models

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “These algorithms incorporate control using Monte Carlo Markov Chain (MCMC) techniques (Hoang et al., 2017; Qin et al., 2020; Mireshghallah et al., 2022), in which a full sequence is initialized and iteratively updated.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “We compare with Mix-and-Match [36], a recent energy-based decoding method with discrete MCMC sampling (Metropolis-Hastings, in particular).”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “…efficient training and inefficient inference; lower text quality; better controllability Guided Strategy [24, 62, 74, 117, 150, 151] [73, 89, 96, 106] Trainable Strategy [25, 59] We list the main characteristics for each categor from a macro perspective, and representative references…”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “…Deng et al., 2020), machine translation (Shen et al., 2004; Lee et al., 2021; Bhattacharyya et al., 2021), constrained generation (Qin et al., 2022; Mireshghallah et al., 2022), and models for speciﬁc attributes like style (Dathathri et al., 2020; Yang and Klein, 2021), length (Li et al., 2017),…”