Training Diffusion Models with Reinforcement Learning

By Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine. 2023

973 citations recorded by the source corpus.

Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective. However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization (DDPO), that are more effective than alternative reward-weighted likelihood approaches. Empirically, DDPO is able to adapt text-to-image diffusion models to objectives that are difficult to express via prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Finally, we show that DDPO can improve prompt-image alignment using feedback from a vision-language model without the need for additional data collection or human annotation. The project's website can be found at http://rl-diffusion.github.io .

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Information Systems, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Information Systems

In Reinforcement Learning

In Generative Models

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, result): “• Black et al. [2] demonstrated that RL fine-tuning can outperform supervised fine-tuning with reward-weighted loss [17] in optimizing the reward, which aligns with our own observations.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “2.”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, positive): “As shown in [12], the iterative denoising process in flow matching models can be formulated as a Markov Decision Process (MDP) ( S , A , ρ 0 , P, R ) .”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “See Figure 37 in Appendix E for more samples. diffusion models on human preference models or other rewards (Lee et al., 2023; Wu et al., 2023b; Dong et al., 2023; Black et al., 2023; Fan et al., 2023).”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “(14) as a multi-step RL approach in the same setting as DDPO and DPOK [7, 13] (Supp.”
DanceGRPO: Unleashing GRPO on Visual Generation (2025, positive): “Following DDPO [18], we formulate the denoising process of the diffusion model and rectified flow as a Markov Decision Process (MDP): where c is the prompt, and π ( a t | s t ) is the probability from z t to z t − 1 .”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “This may also suggest that we can use RAFT in a more general sense beyond the alignment scenario (e.g. boost the model performance in mathematics).”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “_Additionally , _there _are _two _cars _park ed _in _the MLLM allows it to be adept at a variety of vision tasks [2, 25, 45], meanwhile easily handling some complex content comprehension [23] or generation [4, 15].”