DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

By Jaemin Cho, Abhaysinh Zala, Mohit Bansal. IEEE International Conference on Computer Vision, 2022

302 citations recorded by the source corpus.

Recently, DALL-E [45], a multimodal transformer language model, and its variants including diffusion models have shown high-quality text-to-image generation capabilities. However, despite the realistic image generation results, there has not been a detailed analysis of how to evaluate such models. In this work, we investigate the visual reasoning capabilities and social biases of different text-to-image models, covering both multimodal transformer language models and diffusion models. First, we measure three visual reasoning skills: object recognition, object counting, and spatial relation understanding. For this, we propose PaintSkills, a compositional diagnostic evaluation dataset that measures these skills. Despite the high-fidelity image generation capability, a large gap exists between the performance of recent models and the upper bound accuracy in object counting and spatial relation understanding skills. Second, we assess the gender and skin tone biases by measuring the gender/skin tone distribution of generated images across various professions and attributes. We demonstrate that recent text-to-image generation models learn specific biases about gender and skin tone from web image-text pairs. We hope our work will help guide future progress in improving text-to-image generation models on visual reasoning skills and learning socially unbiased representations.1

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/212732c649d84382f4e74ca047b13f3c835591d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, background): “Most literature on text-to-visual generation [7, 25, 27, 65, 87] primarily focuses on generating objects, attributes, relations, and scenes.”
Unified Concept Editing in Diffusion Models (2023, background): “Diffusion image generation models have been found to be prone to social and cultural biases [7,24,43], even exaggerating and amplifying societal stereotypes beyond simple imbalances in the training data [4,11], although quantifying ampliﬁcation can be subtle [36].”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “Following the definitions from PaintSkill [7], we ensure the spatial relation of the two boxes belong to (left, right, above, below) .”