Investigating object compositionality in Generative Adversarial Networks

By Sjoerd van Steenkiste, Karol Kurach, J. Urgen Schmidhuber, Sylvain Gelly. 2018

49 citations recorded by the source corpus.

Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work, we investigate object compositionality as an inductive bias for Generative Adversarial Networks (GANs). We present a minimal modification of a standard generator to incorporate this inductive bias and find that it reliably learns to generate images as compositions of objects. Using this general design as a backbone, we then propose two useful extensions to incorporate dependencies among objects and background. We extensively evaluate our approach on several multi-object image datasets and highlight the merits of incorporating structure for representation learning purposes. In particular, we find that our structured GANs are better at generating multi-object images that are more faithful to the reference distribution. More so, we demonstrate how, by leveraging the structure of the learned generative process, one can 'invert' the learned generative model to perform unsupervised instance segmentation. On the challenging CLEVR dataset, it is shown how our approach is able to improve over other recent purely unsupervised object-centric approaches to image generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “Our approach is orthogonal to methods using adversarial training [47–49] or contrastive learning [46] for object discovery: utilizing Slot Attention in such a setting is an interesting avenue for future work.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “For example, van Steenkiste et al. (2020) demonstrates how a (spatial) mixture model that combines the output of multiple deep neural networks is able to learn to generate images as compositions of individual objects and a background (see also Nguyen-Phuoc et al., 2020; Ehrhardt et al., 2020;…”
Generative Adversarial Transformers (2021, methodology): “Most related to our work are certain GAN models for conditional and unconditional visual synthesis: A few methods [21, 33, 56, 65] utilize multiple replicas of a generator to produce a set of image layers, that are then combined through alpha-composition.”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “[56] found that in images with multiple salient objects, FID focuses on only one of them and hypothesized that the metric could be easily fooled by a generative model that focuses on some image statistics (e.”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “Inspired by the analysis-by-synthesis approach for visual perception discussed in cognitive science [44], recent work [3, 7, 11, 36] propose structured latent space models to explain and synthesize images as sets of constituent components which are individually represented using VAEs [19] or GANs [1”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “For image synthesis, a few works aim at object-level compositionality such as adding a pair of sunglasses to a face [51, 3, 58].”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, background): “Regarding this, there have been a number of recent approaches that can (only) generate 3D scene images via object-centric compositional rendering (Nguyen-Phuoc et al., 2020; van Steenkiste et al., 2020; Ehrhardt et al., 2020).”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Recent works therefore investigate how the synthesis process can be controlled at the object-level [3,4,7,18,19,26,45,86,90].”