ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

By Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, Yuxiao Dong. Neural Information Processing Systems, 2023

1,142 citations recorded by the source corpus.

We present a comprehensive solution to learn and improve text-to-image models from human preference feedback. To begin with, we build ImageReward -- the first general-purpose text-to-image human preference reward model -- to effectively encode human preferences. Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer. Both automatic and human evaluation support ReFL's advantages over compared methods. All code and datasets are provided at \url{https://github.com/THUDM/ImageReward}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1b2355c3c674b26a977768a91a164384ad51bbb1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Training Diffusion Models with Reinforcement Learning (2023, background): “(2023), DPOK only considers a single preference-based reward function (Xu et al., 2023); additionally, their work studies KL-regularization and primarily focuses on training a different diffusion model for each prompt.”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, positive): “For the reward model, we use ImageReward [43] which is trained on a large dataset comprised of human assessments of images.”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “…to align pretrained T2I models with human preferences follow five main directions: (1) direct fine-tuning with differentiable rewards [30, 31, 32, 33]; (2) Reward Weighted Regression (RWR) [34, 35, 36, 37]; (3) Direct Preference Optimization (DPO) and variants [38, 39, 14, 40, 41, 42, 43,…”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “Specifically, PickScore [33] and HPSv2 [83] finetune the CLIP-H model, and ImageReward [84] finetunes the BLIPv2, using costly human feedback from either random web users or expert annotators.”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, neutral): “We further examine the influence of the update frequency for the fake diffusion model µ fake in Figure 9.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, positive): “ReFL (Xu et al., 2023) uses reward function gradients to fine-tune a diffusion model.”
DanceGRPO: Unleashing GRPO on Visual Generation (2025, neutral): “…remains primitive compared with LLMs. Key approaches in this area include: (1) Direct Policy Optimization (DPO)-style [12, 14, 41, 52, 53] methods, (2) direct backpropagation with reward signals [54], such as ReFL [9], and (3) policy gradient-based methods, including DPOK [19] and DDPO [18].”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “We use five mainstream evaluation metrics to evaluate the performance of our Sana, namely FID, Clip Score, GenEval (Ghosh et al., 2024), DPG-Bench (Hu et al., 2024), and ImageReward (Xu et al., 2024), comparing it with SOTA methods.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “Additionally, we employ Aesthetic-V2 to filter out images with low aesthetic scores.”
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (2023, methodology): “Similarly, the Fr´echet Inception Distance (FID) [5] is still the main metric used for model evaluation.”