A Distributional Approach to Controlled Text Generation

By Muhammad Khalifa, Hady ElSahar, Marc Dymetman. 2020

140 citations recorded by the source corpus.

We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both “pointwise” and “distributional” constraints over the target LM — to our knowledge, the first model with such generality — while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence.1

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “Such distributional bias becomes apparent only upon analysing multiple predictions and requires distinct forms of evaluation and correction (Khalifa et al., 2021).”
Challenges in Detoxifying Language Models (2021, background): “Distinct n-gram based filtering: PPLM can occasionally lead to degenerate samples, as noted in the work of Khalifa et al. (2020).”
Pretraining Language Models with Human Preferences (2023, background): “, 2019), alternative finetuning objectives included divergence from a target distribution (Khalifa et al., 2021; Korbak et al., 2022a) or supervised finetuning on data generated by other LMs (Scheurer et al.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Modeling choices (Hooker, 2021) and the lack of control, e.g., Khalifa et al. (2021), can make matters worse by overamplifying existing data bias (Zhao et al., 2017; Shah et al., 2020).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Concerning the gender debiasing experiments, we can see that all other variants of f -DPG outperform the original KL-DPG explored in Khalifa et al. (2021), with RKL-DPG giving the best results and better matching the pointwise constraint although seemingly at the cost of lower diversity as measured…”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “We investigate the benefits of adding a baseline to the DPG algorithm, on the Generation with Distributional Control (GDC) (Khalifa et al., 2021) framework.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “, generating specific words or phrases [6]) or both [24] can be achieved.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “Similarly, methods for controlling models at inference time, e.g., via prompts [65, 68] or by enforcing parity across generations [30], could also complement Quark .”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, neutral): “This distinction is similar to Khalifa et al. [64]’s “pointwise” and “distributional” constraints and is also referenced in analysis of Gopher [85] and PaLM [26] outputs.”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “Then, following (Khalifa et al., 2021), one could deﬁne π ∗ ( x ) = 1 Z π 0 ( x ) b ( x ) , where b ( x ) = 1 if x contains a curse and 0 otherwise.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “In addition, neural LM generation is hard to control, although there are some first steps in this direction (Khalifa et al., 2021; Smith et al., 2020b).”
Controlling Conditional Language Models without Catastrophic Forgetting (2021, methodology): “Extending the approach of Khalifa et al. (2021) such a conditional EBM represents the ideal behaviour of a generative model, given context c, as the distribution that incorporates the control objectives while remaining as close as possible to the original distribution to prevent catastrophic…”
Gradient-based Constrained Sampling from Language Models (2022, methodology): “While this work and the work described above focuses on controlling attributes of individual text outputs, in related work, Khalifa et al. (2021); Kor-bak et al. (2022) develop approaches for distributional control of various attributes in generated text corpora.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Enhance coherence [93, 132], preserve factuality [33, 46, 130, 172], improve controllablity [41, 111, 179].”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “Recent work has thus made attempts to train text-based EBMs each for specific tasks (Deng et al., 2020; Khalifa et al., 2020; Parshakova et al., 2019; Hu et al., 2018).”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “…training and inefficient inference; lower text quality; better controllability Guided Strategy [24, 62, 74, 117, 150, 151] [73, 89, 96, 106] Trainable Strategy [25, 59] We list the main characteristics for each categor from a macro perspective, and representative references for each subcategory.”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, background): “There has been some speculation as to the possible effects of different steps of the RLHF pipeline on diversity (janus, 2022), and some work has shown a decrease in diversity from RLHF (Khalifa et al., 2021; Perez et al., 2022).”
An approximate sampler for energy-based models with divergence diagnostics (2022, methodology): “Controlling the distribution of these language models to accommodate human preferences can be difficult, but EBMs are a promising way to achieve this (Khalifa et al., 2021).”
Energy-Based Models for Code Generation under Compilability Constraints (2021, methodology): “This solution doesn’t have an explicit notion of the optimal policy and often has hard time balancing between the reward term and the KL penalty term, leading to instability in training (Khalifa et al., 2021).”