Generating Multiple Objects at Spatially Distinct Locations

By Tobias Hinz, Stefan Heinrich, Stefan Wermter. 2019

107 citations recorded by the source corpus.

Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. Furthermore, conditional GANs allow us to control the image generation process through labels or even natural language descriptions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. This is especially true for images that should contain multiple distinct objects at different spatial locations. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image background and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image background.

Subfields: Cancer Research, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “In [44], [45], locations of multiple objects are controlled in text-to-image generation by adding an extra object pathway in both the generator and discriminator.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Others have proposed hierarchical models that explicitly generate different objects after inferring semantic layouts [18, 16, 22].”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Several other works propose using hierarchical models that generate images by explicitly modeling the location and semantics of objects [46, 70, 71, 72].”
Image Synthesis From Reconfigurable Layout and Style (2019, background): “In [16, 12, 11, 21], layout and object information is utilized in text-to-image generation.”
Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge (2019, methodology): “Another popular approach has been to provide more information for image generation [9, 7, 11].”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “This includes conditional GANs that use segmentation masks [40, 49], scene graphs [22], object labels, key points or bounding boxes [18, 42], which have shown impressive results for natural image datasets.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Recently, control of the location of individual objects within an image was made possible (Hinz et al., 2019) by adding a pathway in an iterative manner and applying them at different locations specified by the bounding boxes to both the generator and the discriminator.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Therefore, jointly conditioning on text and layout has been studied in [7, 13, 17] as well as predicting layout from text [15].”
Investigating object compositionality in Generative Adversarial Networks (2018, unknown): “, 2019), or their properties (Hinz et al., 2019).”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Their method is built upon AttnGAN-OP [97] and consists of three key components: i) a QA encoder that takes QA pairs as input to produce global and local representations, ii) a QA-conditioned GAN that takes the representations from the QA encoder to produce the image in a two-stage process, and…”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “Our approach relies on an object pathway similar to [3], which iteratively attends to all objects that need to be generated given the current image description and which is conditioned on the image description and a one-hot label describing the object’s class (e.g. “person”).”
Collaging Class-specific GANs for Semantic Image Synthesis (2021, background): “Others [36, 14, 21, 10, 13, 29, 31] model an image via different components, e.”
Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis (2019, background): “*: Inception score of AttnGAN in paper [37] (AttnGAN+Object Pathways) is 23.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “pathways in both a generator and discriminator to control object locations explicitly [220].”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, background): “It is important to note that generative models [19] can be prohibitively challenging to train and optimize on large-scale real-world datasets.”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, unknown): “In addition, the recent AttnGAN [35] and its subsequent improvements [10], [13], [17], [22], [31] adopt the word-level attention mechanism to enhance the local semantic matching between word features and local image features.”
On the Evaluation of Conditional GANs (2019, background): “Image-to-image conditional generation has also been framed as a mask-to-image or bounding box-to-image task [14, 13, 29, 54].”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “Layout-to-image generation also serves as a useful subpro-cess in text-to-image translation [11, 12, 13, 20] and image generation from scene graphs [1, 15].”