Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

By Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy. Neural Information Processing Systems, 2023

1,061 citations recorded by the source corpus.

The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/cf694df964caa156ec306b45d3a3127533cb458f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DanceGRPO: Unleashing GRPO on Visual Generation (2025, positive): “We also take the metric like the Pick-a-Pic [39] and GenEval [27] to evaluate our method.”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, background): “Learning from human feedback (LHF) has proven to be an effective means to overcome these limitations [13, 17, 42, 43].”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “…against five popular method types: (1) VLM-based metrics (CLIPScore [21] and BLIPv2Score [40]); (2) VLMs finetuned on human feedback (PickScore [33], ImageReward [84], and HPSv2 [83]); (3) visual programming methods (VisProg [20], ViperGPT [73], and VPEval [9]); (4) divide-and-conquer methods…”
Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023, methodology): “For example, 34.9% of instances in Pick-a-Pic has a prompt that repeats for more than 50 times, these repeated prompts only take up 4.7% of total unique prompts.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “We employed t-SNE[57] to visualize the prompt distribution of training sets of PickScore and ImageReward.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, positive): “We scale up DRaFT to train on hundreds of thousands of prompts and show that it improves Stable Diffusion on human preference rewards such as PickScore (Kirstain et al., 2023) and Human Preference Score v2 (Wu et al., 2023a).”
Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “Caption-aware human preference scoring models are trained on generation preference datasets [24, 55, 58], but the impact of these reward models to the generative space has been limited.”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “(2) Model-based rewards, such as the human preference Pickscore [19] reward.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “Since standard performance metrics for generative models [45, 16] measure practically relevant quality image aspects as aesthetics and scene composition only insufficiently [27, 38] we use human evaluation to compare our model against the current state-of-the-art in text-to-image synthesis.”
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, background): “This aligns with and further backs the ﬁndings of Kirstain et al. [23].”
PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (2023, background): “Recent studies such as SDXL (Podell et al., 2023) and Pick-a-pic (Kirstain et al., 2023) have presented evidence suggesting that the COCO zero-shot FID is negatively correlated with visual aesthetics.”