Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis

By Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, Hongsheng Li. arXiv.org, 2023

863 citations recorded by the source corpus.

Recent text-to-image generative models can generate high-fidelity images from text inputs, but the quality of these generated images cannot be accurately evaluated by existing evaluation metrics. To address this issue, we introduce Human Preference Dataset v2 (HPD v2), a large-scale dataset that captures human preferences on images from a wide range of sources. HPD v2 comprises 798,090 human preference choices on 433,760 pairs of images, making it the largest dataset of its kind. The text prompts and images are deliberately collected to eliminate potential bias, which is a common issue in previous datasets. By fine-tuning CLIP on HPD v2, we obtain Human Preference Score v2 (HPS v2), a scoring model that can more accurately predict human preferences on generated images. Our experiments demonstrate that HPS v2 generalizes better than previous metrics across various image distributions and is responsive to algorithmic improvements of text-to-image generative models, making it a preferable evaluation metric for these models. We also investigate the design of the evaluation prompts for text-to-image generative models, to make the evaluation stable, fair and easy-to-use. Finally, we establish a benchmark for text-to-image generative models using HPS v2, which includes a set of recent text-to-image models from the academic, community and industry. The code and dataset is available at https://github.com/tgxs002/HPSv2 .

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d8008ace077d72ccd277be795c720e9381623c10

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “9, Compared to T5-XXL, Gemma-2-2B has an inference speed that is 6 × faster, while the performance of Gemma-2B is comparable to T5-XXL in terms of Clip Score, FID, GenEval, and HPSv2 (Wu et al., 2023).”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “Specifically, PickScore [33] and HPSv2 [83] finetune the CLIP-H model, and ImageReward [84] finetunes the BLIPv2, using costly human feedback from either random web users or expert annotators.”
Emu3: Next-Token Prediction is All You Need (2024, background): “1 [90], MPS [105], and the LAION Aesthetics score [43].”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, neutral): “Additionally, we employ Aesthetic-V2 to filter out images with low aesthetic scores.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “The results above the separator are taken from Wu et al. (2023a); the results below were obtained from our implementations and experiments.”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “Caption-aware human preference scoring models are trained on generation preference datasets [24, 55, 58], but the impact of these reward models to the generative space has been limited.”
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, methodology): “Inspired by past work in NLP, several new methods evaluate generated images using models trained on direct human preference data [21, 41, 42].”