Semantic Object Accuracy for Generative Text-to-Image Synthesis

By Tobias Hinz, Stefan Heinrich, Stefan Wermter. 2019

188 citations recorded by the source corpus.

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g., whether an image generated from “a car driving down the street” contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We compare L AFITE against a series of competitive systems: AttnGAN [51], Obj-GAN [23], DM-GAN [57], OP-GAN [13], DF-GAN [42] and XMC-GAN [54].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Further details of SOA can be found in [17].”
Unifying Multimodal Transformer for Bi-directional Image and Text Generation (2021, methodology): “We do not use Inception score (IS) [41], since it overfits within the context of text-to-image generation and can be manipulated to achieve much higher scores using simple tricks [5, 15].”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “Text-to-Image Synthesis There also exists work on generation of images conditioned on text [16, 24, 62, 36, 37, 47, 48, 52, 67, 14, 19, 54, 27], also known as text-to-image synthesis.”
Blended Diffusion for Text-driven Editing of Natural Images (2021, background): “Recent works that leverage the tremendous progress in vision-language models and datadriven image generation have demonstrated that text-based interfaces for image creation and manipulation are now finally within reach [12, 23, 29, 30, 38, 39, 41, 47, 51, 57].”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “Layout-to-image generation also serves as a useful subpro-cess in text-to-image translation [11, 12, 13, 20] and image generation from scene graphs [1, 15].”
CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “Speciﬁcally, it includes two sub-metrics: SOA-C (average recall w.r.t. object category) and SOA-I (average recall w.r.t. image sample), which are deﬁned as: where C and I C refer to the set of categories and set of images in the category c respectively.”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “SOA [19] and DALL-Eval [6] employ object detection models to determine if objects, attributes, and relations in the text input are in the generated image.”
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, background): “Many of these are reference-based: BLEU [28] and CIDEr [40] can be used to evaluate generations by performing captioning and comparing to a set of reference captions [17, 18], while R-Precision [29, 43] measures text recall scores from a reference dataset of captions.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “However, R-prec. often fails for COCO images, in which a high similarity can be assigned to wrong captions which mention the global background color (e.g., “snow” for images with white background) or objects that appear in the center [108].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “Numerous works [11, 16, 42, 25, 26, 30, 31, 34, 49, 9, 13, 36] significantly advanced the generation of images conditioned on plain text, known as text-to-image synthesis.”
Modeling Image Composition for Complex Scene Generation (2022, background): “Though the proposed TwFA achieves impressive performance on (few-shot) L2I tasks, it relies on the composition prior from given layouts which may not be available in other complex scene generation tasks, e.g. , textual descriptions [29] and scene graphs [6].”
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers (2022, methodology): “4 SOA only focuses on the existence of objects, which makes it not well suited to evaluate object attributes and relation between objects [20, 27].”
DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (2022, methodology): “Semantic Object Accuracy (SOA) [29] measures whether an object detector can detect an object described in the text from a generated image.”
Lafite2: Few-shot Text-to-Image Generation (2022, methodology): “We compare our methods with different fully-supervised methods including AttnGAN [39], Obj-GAN [17], DM-GAN [45], OP-GAN [10], DF-GAN [34], XMC-GAN [41], L AFITE [44] and Make-A-Scene [7].”