Diffusion Model Alignment Using Direct Preference Optimization

By Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq R. Joty, Nikhil Naik. Computer Vision and Pattern Recognition, 2023

819 citations recorded by the source corpus.

Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' preferences. In contrast to LLMs, human preference learning has not been widely explored in text-to-image diffusion models; the best existing approach is to fine-tune a pretrained model using carefully curated high quality images and captions to improve visual appeal and text alignment. We propose Diffusion-DPO, a method to align diffusion models to human preferences by directly optimizing on human comparison data. Diffusion-DPO is adapted from the recently developed Direct Preference Optimization (DPO) [36], a simpler alternative to RLHF which directly optimizes a policy that best satisfies human preferences under a classification objective. We re-formulate DPO to account for a diffusion model notion of likelihood, utilizing the evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with Diffusion-DPO. Our fine-tuned base model significantly outperforms both base SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement model in human evaluation, improving visual appeal and prompt alignment. We also develop a variant that uses AI feedback and has comparable performance to training on human preferences, opening the door for scaling of diffusion model alignment methods.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f5275c61736781d236abe6700b822f1ea62f982e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, neutral): “RL has proven to be an effective learning paradigm in text and image processing [Schulman et al., 2017, Rafailov et al., 2024, Sutton et al., 1999, Esser et al., 2024, Wallace et al., 2023].”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “For better human preference alignment, we finetune our models via Diffusion DPO ([61]), an adaption of the Direct Preference Optimization (DPO) [40] technique to diffusion models.”
DanceGRPO: Unleashing GRPO on Visual Generation (2025, neutral): “…limitations: ReFL [9–11] relies on differentiable reward models, which introduce VRAM inefficiency in video generation and require several extensive engineering efforts, while DPO variants (Diffusion-DPO [12, 13], Flow-DPO [14], OnlineVPO [15]) achieve only marginal visual quality improvements.”
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “Recently, this method has been adapted to preference finetuning of text-to-image diffusion models (Wallace et al., 2023).”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “We compare Flow-GRPO with several alignment methods: supervised fine-tuning (SFT), Flow-DPO [14, 39], and their online variants.”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, methodology): “Furthermore, our methods are optimized for distribution matching; incorporating human feedback or other reward functions could further enhance performance [17,73].”
Qwen-Image Technical Report (2025, neutral): “Then, inspired by previous work (Wallace et al., 2024), we construct the DPO objective based on the flow matching training criterion, which can be formulated as follows: where Diff policy and Diff ref denote the preference differences computed by the policy model and the reference model,…”