A Case for Object Compositionality in Deep Generative Models of Images

By Sjoerd van Steenkiste, Karol Kurach, Sylvain Gelly. 2018

20 citations recorded by the source corpus.

Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work we propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition. This provides a way to efficiently learn a more accurate generative model of real-world images, and serves as an initial step towards learning corresponding object representations. We evaluate our approach on several multi-object image datasets, and find that the generator learns to identify and disentangle information corresponding to different objects at a representational level. A human study reveals that the resulting generative model is better at generating images that are more faithful to the reference distribution.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[47] Sjoerd van Steenkiste, Karol Kurach, and Sylvain Gelly.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Finally, while this paper focuses on evaluating the benefits of disentangled features, these are complementary to recent work that focuses on the unsupervised “disentangling” of images into compositional primitives given by object-like representations [17, 23, 24, 22, 60, 74, 75].”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, background): “[26] can generate scenes with an arbitrary number of components.”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, result): “Similar to previous works (van Steenkiste et al., 2018; Crawford and Pineau, 2019; Burgess et al., 2019; Greﬀ et al., 2019; Engelcke et al., 2020; Lin et al., 2020), for each layer n , we compute a transparency map α n,q = w n,q ˆ m n,q , where is pixel-wise multiplication.”
Unsupervised Object Segmentation by Redrawing (2019, background): “2018 [52] advocates for a generative framework that accounts for relationship between objects.”
Compositional Visual Generation with Energy Based Models (2020, background): “Another approach to incorporate the compositionality is to spatially decompose an image into a collection of objects, each object slot occupying some pixels of the image deﬁned by a segmentation mask [28, 6].”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, methodology): “Recently, unsupervised methods [2, 12, 13, 26, 50, 55] learned object disentanglement for multi-object scenes on simpler synthetic datasets (single-colour objects, simple lighting, and material).”