Direct Preference Optimization: Your Language Model is Secretly a Reward Model

By Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Neural Information Processing Systems, 2023

9,702 citations recorded by the source corpus.

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Subfields: Computer Networks and Communications, Economics and Econometrics, General Decision Sciences, Large Language Models, Alignment & RLHF

View Paper: https://arxiv.org/abs/2305.18290

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

  1. Ion Stoica
  2. Matei A. Zaharia
  3. Dan Hendrycks
  4. Joseph E. Gonzalez
  5. Nando de Freitas
  6. Percy Liang
  7. Dacheng Tao
  8. Mantas Mazeika

In Economics and Econometrics

  1. Rafael Rafailov
  2. Steven Basart
  3. Damai Dai
  4. Dylan Hadfield-Menell
  5. Dacheng Tao
  6. Peiyi Wang
  7. Zhihong Shao
  8. Yuan Yao

In General Decision Sciences

  1. Rafael Rafailov
  2. Dylan Hadfield-Menell
  3. Carlos Guestrin
  4. Saining Xie
  5. Penghao Wu
  6. Stefano Ermon
  7. Anca Dragan
  8. Kyle Richardson

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Yejin Choi
  3. Zhiyuan Liu
  4. Danqi Chen
  5. Percy Liang
  6. Hinrich Schütze
  7. Maosong Sun
  8. Arman Cohan

In Alignment & RLHF

  1. Rafael Rafailov
  2. Tianyi Zhang
  3. Dylan Hadfield-Menell
  4. David Krueger
  5. Yu Meng
  6. Percy Liang
  7. Yejin Choi
  8. Carlos Guestrin

Citations in the corpus