A General Theoretical Paradigm to Understand Learning from Human Preferences

By Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, D. Guo, Daniele Calandriello, Michal Valko, Rémi Munos. International Conference on Artificial Intelligence and Statistics, 2023

1,063 citations recorded by the source corpus.

The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of $\Psi$PO) and to identify their potential pitfalls. We then consider another special case for $\Psi$PO by setting $\Psi$ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/f3460dc3ae5cfd41099d576a3bb77411e1fc2e3f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “IPO [6] is a theoretically grounded approach method that avoids DPO’s assumption that pairwise preferences can be replaced with pointwise rewards.”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “There is also a line of works studying direct preference learning algorithms [Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2023; Tang et al., 2024], which bypasses traditional reward modeling to learn directly from preference datasets in a supervised manner (hence the name direct…”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “Preference tuning further improves responses of LLMs using reinforcement learning human feedback (RLHF) [5] or preference optimization [2, 19, 23, 44] based on a preference dataset.”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, background): “Many offline alternatives have also shown promises for better learning from feedbacks, which includes SLiC-HF [Zhao et al., 2023], DPO [Rafailov et al., 2023], IPO [Azar et al., 2023], KTO [Ethayarajh et al., 2023], ReST [Gulcehre et al., 2023] and RSO [Liu et al., 2024].”
ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “Azar et al. (2023) prevented potential overfitting problems in DPO through identity preference optimization (IPO).”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, methodology): “Since then, several training algorithms have been proposed to efficiently improve the alignment, including direct preference optimization (DPO) (Rafailov et al., 2023), identity preference optimization (IPO) (Azar et al., 2023), and Kahneman-Tversky optimization (KTO) (Etha-2 yarajh et al., 2023).”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “Similar to DPO, there are also other works aiming to optimize the LLMs directly from the preference data, including (Zhao et al., 2023; Azar et al., 2023), and has sparked considerable debate on whether reward modeling, as well as RL, is necessary for alignment.”
Model Alignment as Prospect Theoretic Optimization (2024, background): “Other preference-based losses include CPO (Xu et al., 2024) and IPO (Azar et al., 2024).”
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, background): “Azar et al. [2023] aims to understand the theoretical underpinnings of RLHF and DPO.”
Process Reinforcement through Implicit Rewards (2025, positive): “Implicit rewards are broadly adopted in LLM alignment (Rafailov et al., 2023; Chen et al., 2024b; Azar et al., 2024; Ethayarajh et al., 2024; Rosset et al., 2024; Chen et al., 2024a).”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “IPO (Azar et al., 2024) Meng et al., 2024) introduces a reference-free objective that mitigates length bias by maximizing the gap between the preferred and dispreferred responses’ average log-likelihood.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “In view of the above issues of PPO, there is an innovative line of work that directly learns from human preference datasets without explicitly constructing a reward function (Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2023).”
Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “For Al-pacaEval 2.0 LC performance, we observe that only IPO clearly outperforms BCO.”