Aligning Text-to-Image Models using Human Feedback

By Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Shixiang Shane Gu. arXiv.org, 2023

471 citations recorded by the source corpus.

Deep generative models have shown impressive results in text-to-image synthesis. However, current text-to-image models often generate images that are inadequately aligned with text prompts. We propose a fine-tuning method for aligning such models using human feedback, comprising three stages. First, we collect human feedback assessing model output alignment from a set of diverse text prompts. We then use the human-labeled image-text dataset to train a reward function that predicts human feedback. Lastly, the text-to-image model is fine-tuned by maximizing reward-weighted likelihood to improve image-text alignment. Our method generates objects with specified colors, counts and backgrounds more accurately than the pre-trained model. We also analyze several design choices and find that careful investigations on such design choices are important in balancing the alignment-fidelity tradeoffs. Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1e4b6567ff66de6cdcbe58c863538241e2f62b45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (2023, background): “Lee et al. [8] consider three simple categories of challenges (count, color, and background), enumerating through templates (e.g. “[number] dogs”) to synthetically create prompts.”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, unknown): “• We study incorporating KL regularization into supervised fine-tuning of diffusion models, which can mitigate some failure modes (e.g., generating over-saturated images) in [17].”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, background): “Direct multi-concept generation using text prompts alone faces challenges such as missing objects and attribute binding [6, 31, 32, 33, 34].”
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, unknown): “Some recent works attempt to align text-to-image models with human feedback [30, 31].”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, unknown): “…follow five main directions: (1) direct fine-tuning with differentiable rewards [30, 31, 32, 33]; (2) Reward Weighted Regression (RWR) [34, 35, 36, 37]; (3) Direct Preference Optimization (DPO) and variants [38, 39, 14, 40, 41, 42, 43, 44, 45, 46]; (4) PPO-style policy gradients [47, 48, 49,…”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “It is also possible to apply RLHF to general generative models, like the diffusion model (Hao et al., 2022; Wu et al., 2023; Lee et al., 2023; Dong et al., 2023).”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “Compositional Image Generation Recent studies have shown that text-to-image generation (T2I) models suffer from compositional issues such as missing objects, incorrect spatial relations, and incorrect attributes [24, 2].”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Incorporating human feedback into models in this way has been used to improve summarization (Stiennon et al., 2020; Wu et al., 2021), question answering (Nakano et al., 2021), image models (Lee et al., 2023) and instruction following in general (Ouyang et al., 2022).”
Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023, background): “Recent research in this field has attempted to improve the image quality of diffusion models [24, 8], but the evaluation of these improvements remains challenging due to the lack of appropriate metrics and benchmarks.”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, background): “Models are expected to understand specific visual control signals like colors, counts, and backgrounds [21] more accurately after alignment.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [74, 32, 22, 59, 55, 13, 56, 40], numeric ratings [44, 31], or execution traces [13].”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, background): “Despite the recent [21]’s attempt to leverage reward-witnessed training in a closed domain, the vanilla RLHF is proved more data-efficient, effective, and generalizable in real environments.”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, methodology): “[12] and [21] use similar methodology to improve text-to-image models, which are highly related to our work.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “Inspired by RLHF, supervised and reinforcement learning methods have been developed to train”
Training Diffusion Models with Reinforcement Learning (2023, unknown): “[25] describe a single-round version of this procedure for diffusion models, but in general this approach can be performed for multiple rounds of alternating sampling and training, leading to a simple RL method.”