Direct Preference Optimization: Your Language Model is Secretly a Reward Model
By Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Neural Information Processing Systems, 2023
9,702 citations recorded by the source corpus.
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
Subfields: Computer Networks and Communications, Economics and Econometrics, General Decision Sciences, Large Language Models, Alignment & RLHF
View Paper: https://arxiv.org/abs/2305.18290
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Ion Stoica
- Matei A. Zaharia
- Dan Hendrycks
- Joseph E. Gonzalez
- Nando de Freitas
- Percy Liang
- Dacheng Tao
- Mantas Mazeika
In Economics and Econometrics
- Rafael Rafailov
- Steven Basart
- Damai Dai
- Dylan Hadfield-Menell
- Dacheng Tao
- Peiyi Wang
- Zhihong Shao
- Yuan Yao
In General Decision Sciences
- Rafael Rafailov
- Dylan Hadfield-Menell
- Carlos Guestrin
- Saining Xie
- Penghao Wu
- Stefano Ermon
- Anca Dragan
- Kyle Richardson
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Zhiyuan Liu
- Danqi Chen
- Percy Liang
- Hinrich Schütze
- Maosong Sun
- Arman Cohan
In Alignment & RLHF
- Rafael Rafailov
- Tianyi Zhang
- Dylan Hadfield-Menell
- David Krueger
- Yu Meng
- Percy Liang
- Yejin Choi
- Carlos Guestrin
Citations in the corpus
- Mixtral of Experts (2024, methodology): “We also present Mixtral 8x7B – Instruct, a chat model fine-tuned to follow instructions using supervised fine-tuning and Direct Preference Optimization [25].”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “• We provide a unified paradigm to understand different methods, such as RFT, DPO, PPO, and GRPO.”
- A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Zephyr enhances intent alignment through the application of distilled direct preference optimization (dDPO).”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Batch-Constrained deep Q-learning (BCQ) [294] presents an approach aimed at enhancing off-policy learning by constraining the action space.”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “The whole content structural arrangement is shown in Figure 5.”
- OLMo: Accelerating the Science of Language Models (2024, methodology): “Additionally, we find that OLMo-7B outperforms most other chat variants after both initial instruction tuning (OLMo+SFT) and additional preference alignment (OLMo+SFT+DPO), highlighting both the strength of OLMo-7B as a base model and the strength of the T ÜLU mix used to perform adaptation training”
- Statistical Rejection Sampling Improves Preference Optimization (2023, methodology): “The AutoSxS has been demonstrated as effective and consistent in Rafailov et al. (2023) using GPT-4 as zero-shot rater.”
- Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “Concurrent to our work, (Rafailov et al., 2023) proposed a method called DPO (Direct Preference Optimization) that can make use of offline data to align an LM with human preferences.”
- UltraFeedback: Boosting Language Models with High-quality Feedback (2023, neutral): “, 2023b), design novel algorithms (Gulcehre et al., 2023; Rafailov et al., 2023; Song et al., 2023a) to utilize preference data more efficiently, or align LLMs with various types of feedbacks (Yao et al.”
- A Survey on Large Language Models for Code Generation (2024, neutral): “Figure 6 illustrates the LLM benchmark structure that includes the integration of metrics within this evaluation framework.”
- LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “We utilize the trainers of Transformers (Wolf et al., 2020) for pre-training and SFT, while adopting the trainers of TRL (von Werra et al., 2020) for RLHF and DPO.”
- Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “An early and influential application of RL in LLMs is the Reinforcement Learning from Human Feedback (RLHF) [51, 52, 53, 54], which focuses on aligning LLMs to human preferences.”
- SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “For alignment techniques, TRL 3 is introduced to extend base trainer class from Transformers and implements specific meth-ods to support techniques such as Direct Preference Optimization (DPO)[49], Optimized Reward Policy Optimization (ORPO)[24], and Knowledge Transfer Optimization (KTO)[16].”
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “This formulation is widely studied in practice (Ziegler et al., 2019; Wu et al., 2021a; Ouyang et al., 2022; Rafailov et al., 2023; Liu et al., 2023a), and our paper aims to study its theoretical property.”
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “…et al., 2020; Rafailov et al., 2023) have been developed; however, these methods can be vulnerable to jailbreaks (Wei et al., 2023; Chao et al., 2023; Yao et al., 2023a; Yuan et al., 2023) and adversarial attacks (Wallace et al., 2019; Guo et al., 2021; Jones et al., 2023; Zou et al., 2023b).”
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, neutral): “To simplify RL-based tuning, direct optimization methods such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) and SimPO (Meng et al., 2024) have been proposed.”
- A survey on multimodal large language models (2023, neutral): “…and previous reasoning history [14,65,75,76,82,83] Training-free Few-shot Chameleon [75], HuggingGPT [80], MM-REACT [14], VISPROG [76] Zero-shot CAT [84], CaFo [90], Visual ChatGPT [83], IdealGPT [91], ChatCaptioner [92], PointCLIP V2 [93], SMs [94] Finetuning GPT4Tools [34] Functions (§3.4.3)”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “Yet, recent advancements in LLMs [32, 97, 103, 147] and some in MLLMs [132, 137] suggest that reinforcement learning from human or environmental feedback can further improve models, potentially surpassing the limits of supervised fine-tuning, especially in decision-making abilities.”