DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models

By Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, Kimin Lee. Neural Information Processing Systems, 2023

460 citations recorded by the source corpus.

Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an RL problem, and updating the pre-trained text-to-image diffusion models using policy gradient to maximize the feedback-trained reward. Our approach, coined DPOK, integrates policy optimization with KL regularization. We conduct an analysis of KL regularization for both RL fine-tuning and supervised fine-tuning. In our experiments, we show that DPOK is generally superior to supervised fine-tuning with respect to both image-text alignment and image quality. Our code is available at https://github.com/google-research/google-research/tree/master/dpok.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a553bf27d801d09f667fe121c0ba9632257f364b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “(14) as a multi-step RL approach in the same setting as DDPO and DPOK [7, 13] (Supp.”
DanceGRPO: Unleashing GRPO on Visual Generation (2025, unknown): “…remains primitive compared with LLMs. Key approaches in this area include: (1) Direct Policy Optimization (DPO)-style [12, 14, 41, 52, 53] methods, (2) direct backpropagation with reward signals [54], such as ReFL [9], and (3) policy gradient-based methods, including DPOK [19] and DDPO [18].”
Training Diffusion Models with Reinforcement Learning (2023, background): “Like Lee et al. (2023), DPOK only considers a single preference-based reward function (Xu et al., 2023); additionally, their work studies KL-regularization and primarily focuses on training a different diffusion model for each prompt.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “We therefore experimented with a version of DRaFT-LV that only computes the reward gradient with respect to the input pixels ∇ x r ( x 0 , c ) once.”