Aligning Language Models with Preferences through f-divergence Minimization

By Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, Marc Dymetman. 2023

126 citations recorded by the source corpus.

Aligning language models with preferences can be posed as approximating a target distribution representing some desired behavior. Existing approaches differ both in the functional form of the target distribution and the algorithm used to approximate it. For instance, Reinforcement Learning from Human Feedback (RLHF) corresponds to minimizing a reverse KL from an implicit target distribution arising from a KL penalty in the objective. On the other hand, Generative Distributional Control (GDC) has an explicit target distribution and minimizes a forward KL from it using the Distributional Policy Gradient (DPG) algorithm. In this paper, we propose a new approach, f-DPG, which allows the use of any f-divergence to approximate any target distribution that can be evaluated. f-DPG unifies both frameworks (RLHF, GDC) and the approximation methods (DPG, RL with KL penalties). We show the practical benefits of various choices of divergence objectives and demonstrate that there is no universally optimal objective but that different divergences present different alignment and diversity trade-offs. We show that Jensen-Shannon divergence strikes a good balance between these objectives, and frequently outperforms forward KL divergence by a wide margin, leading to significant improvements over prior work. These distinguishing characteristics between divergences persist as the model size increases, highlighting the importance of selecting appropriate divergence objectives.

Subfields: Artificial Intelligence, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Alignment & RLHF

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “⇣ ⌘ Even if we use the MLE estimate r � of the ground-truth reward function r ⇤ , it is still expensive to estimate the partition function Z ( x ) [17, 15], which makes this representation hard to utilize in practice. is the partition function.”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, methodology): “This umbrella of oversight - commonly referred to as ’alignment’ - remains a hot topic for research (Xu et al., 2020; Ouyang et al., 2022; Bai et al., 2022b; Go et al., 2023; Korbak et al., 2023).”
Model Alignment as Prospect Theoretic Optimization (2024, background): “The surprising success of offline PPO with dummy +1/-1 rewards suggests that—with the right inductive biases—a binary signal of good/bad generations may be sufficient to reach DPO-level performance, even if the offline PPO approach itself was unable to do so past a certain scale (§3.3).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “RL finetuning decreases the diversity of samples produced by a model [Khalifa et al., 2021, Perez et al., 2022a, Glaese et al., 2022, Go et al., 2023] (a phenomenon known as “mode collapse”).”