DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models

By Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, Duen Horng Chau. Annual Meeting of the Association for Computational Linguistics, 2022

493 citations recorded by the source corpus.

With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: https://poloclub.github.io/diffusiondb.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/8fe6a16df99a87f79fc0cb0fbd1af44f79f7885f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In General Social Sciences

Citations in the corpus

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, background): “This practice aids in gaining a comprehensive understanding of the task and helps craft high-quality templates.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “However, it has been shown [Feng et al. 2022; Wang et al. 2022] that images produced by such models do not always faithfully reflect the semantic meaning of the target prompt. and incorrect attribute binding (right).”
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (2023, background): “ImageReward [18] selects prompts and images from the DiffusionDB dataset [16], and collects for them image preference ratings via crowd workers.”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, methodology): “CLIP Score [29] and HPS are computed on prompts from DiffusionDB [45].”
Power Hungry Processing: Watts Driving the Cost of AI Deployment? (2023, unknown): “…study, we chose 10 ML tasks from 5 different modalities: Text-to-category (text classification, token classification, extractive question answering), Text-to-text (masked language modeling, text generation, summarization), Image-to-category (image classification, object detection),…”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “To evaluate the ability of ImageReward to select the more preferred images among large amounts of generated images, we produce another dataset, collecting prompts with 9/25/64 generated images from DiffusionDB, and use different methods to select from those images to get top3 results.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “Human Preference Score v2 (HPSv2; Wu et al. 2023a) is trained on prompts from DiffusionDB (Wang et al., 2023b) and COCO Captions (Chen et al., 2015).”
GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image (2023, background): “DiffusionDB [31] is a large-scale gallery dataset with 1.”
Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023, methodology): “ImageReward [25] introduces a dataset collected with a similar methodology like us, but the text prompts are directly taken from DiffusionDB [23], which suffers from severe bias.”