Quark: Controllable Text Generation with Reinforced Unlearning

By Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, Yejin Choi. 2022

277 citations recorded by the source corpus.

Large-scale language models often learn behaviors that are misaligned with user expectations. Generated text may contain offensive or toxic language, contain significant repetition, or be of a different sentiment than desired by the user. We consider the task of unlearning these misalignments by fine-tuning the language model on signals of what not to do. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property, while not straying too far from the original model. Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model's input, and (iii) using a standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via a KL-divergence penalty. By conditioning on a high-reward token at generation time, the model generates text that exhibits less of the unwanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO (Schulman et al. 2017), while relying only on standard language modeling primitives.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “Since human feedback is costly, several approaches use a scalar reward function as a surrogate of (or alternative to) human feedback (e.g., (Bai et al., 2022a; Liu et al., 2022; Lu et al., 2022; Le et al., 2022a; Welleck et al., 2022)).”
Pretraining Language Models with Human Preferences (2023, background): “…2018; Keskar et al., 2019) has been shown to be successful across tasks as diverse as as controllable language generation (Peng et al., 2018; Dai et al., 2019), mitigating toxicity (Gehman et al., 2020; Xu et al., 2020; Lu et al., 2022) and robotic control (Chen et al., 2021a; Janner et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2022) Misinformation / Factuality Autoregressive factual error control(King et al., 2022; Lu et al., 2022); Non-autoregressive factual error control (Kumar et al.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “Most recent works that tune LMs using RL do so by calculating a reward for all the tokens in the sentence (Wu et al., 2021a; Ouyang et al., 2022; Lu et al., 2022).”
Director: Generator-Classifiers For Supervised Language Modeling (2022, background): “Lu et al. (2022) proposes a way to control text generation with iterative reinforcement to deal with toxic generations or negative sentiment.”
Editing Models with Task Arithmetic (2022, unknown): “[59] Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi.”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “…on retrieval and generation, we train our target LM on task examples augmented with reflection tokens from a critic model offline, with a far lower training cost compared to RLHF. Compared to prior work using control tokens to guide LM generation (Lu et al., 2022; Korbak et al., 2023), S ELF -”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2021), PPO, Quark (Lu et al., 2022) and Self-Correct (Welleck et al.”
Model Alignment as Prospect Theoretic Optimization (2024, methodology): “However, Korbak et al. (2023) found unlikelihood training to be worse than the CSFT baseline we tested in this work, which is among various approaches that convert a binary/discrete signal into a control token (Lu et al., 2022).”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “This framework has been explored to improve the model performance on a variety of natural language processing tasks such as text summarization [40], instruction following [29], question answering [24, 27] and reducing harmfulness [3, 2, 22, 10].”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, unknown): “In fact, sometimes unlearning is used to describe tools designed to improve models by making them forget bad behaviors (Hu et al., 2023; Yao et al., 2023; Lu et al., 2022).”
Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, background): “Discretizing the reward scale and pushing contrastive reward scores farther apart has been a successful strategy in other text ﬁne-tuning tasks (cf. (Lu et al., 2022)).”
Large Language Model Unlearning (2023, unknown): “• Fluency : Following the prior work [Lu et al., 2022], we use fluency (the perplexity of generated text tested on a reference LLM) to measure the quality of outputs.”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “…et al., 2021), summarization (Stiennon et al., 2020; Paulus et al., 2018), machine translation (Shen et al., 2016; Wu et al., 2016), grounded text generation (Ammanabrolu et al., 2021, 2022), controlled text generation (Lu et al., 2022), and prompt generation (Guo et al., 2021; Deng et al., 2022).”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “Second, we found that training on rollouts of more bins led to worse efficiency for reward optimization, and thus opted to train only on rollouts of the top-scoring bin (best-quantile variant in the original paper [39]).”