GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

By Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, Ingmar Posner. 2019

343 citations recorded by the source corpus.

Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “In line with previous works [16, 17, 27], we exclude background labels in ARI evaluation.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2019), GENESIS (Engelcke et al., 2019), and ECON (von Kügelgen et al.”
Towards causal generative scene models via competition of experts (2020, methodology): “We use the same attention network architecture as in Burgess et al. (2019) and the implementation provided by Engelcke et al. (2019).”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, methodology): “Furthermore, both IODINE and GENESIS require storing data for each of the K components, so we were unable to run our experiments on 256 components or greater before running out of memory on our 22GB GPU.”
Stacked Capsule Autoencoders (2019, methodology): “The image is modelled as a spatial Gaussian mixture, similarly to Greff et al., 2019; Burgess et al., 2019; Engelcke et al., 2019.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Recent works therefore investigate how the synthesis process can be controlled at the object-level [3,4,7,18,19,26,45,86,90].”
Scalable Object-Oriented Sequential Generative Models (2019, background): “Scene-mixture models (Greff et al., 2017; Van Steenkiste et al., 2018; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) decompose scenes into a collection of components, each being a full-image level representation.”
Learning Physical Graph Representations from Visual Scenes (2020, background): “Recently, machine learning researchers have developed deep generative models for unsupervised segmentation by capturing relations between scene components [10, 4, 17, 9], with varying degrees of success and generality.”
SCALOR: Generative World Models with Scalable Object Representations (2019, background): “On the other hand, scene-mixture models (Greff et al., 2017; Van Steenkiste et al., 2018; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) decompose scenes into a collection of components, each being a full-image level representation.”
Neural Production Systems (2021, background): “…(Le Roux et al., 2011; Eslami et al., 2016; Greff et al., 2016; Raposo et al., 2017; Van Steenkiste et al., 2018; Kosiorek et al., 2018; Engelcke et al., 2019; Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020a; Ahmed et al., 2020; Goyal et al., 2019; Zablotskaia et al.,…”
Unknown Object Segmentation from Stereo Images (2021, methodology): “In a similar manner, multiple encoder-decoder steps are applied in [19], [20] to refine object-centric representations.”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, methodology): “ This can be seen as an object-speciﬁc image layer. The last step is to composite these N layers into a single image so that occlusion among objects can be properly handled. Similar to previous works [9, 4, 16, 11, 51, 34], for each layer n, we compute a transparency map what n;q from o and odepth, ”
Kubric: A scalable dataset generator (2022, methodology): “While recent models such as IODINE [36], MONet [12], GENESIS [28], and Slot Attention [59] succeed at decomposing simple scenes with uniform textures, decomposing dynamic scenes (i.e., videos) with high visual complexity and complex dynamics remains difficult.”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “In terms of what we consider to present the most opportunity for future work, we note that there are relatively few continuous optimization approaches which seek to learn structured, semantic representations from nonsemantic, high-dimensional data such as video or image data (exceptions include Caus”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “Furthermore, we compare RELATE to several recent GANand VAE-based baselines, including BlockGAN [29], GENESIS [7] and OCF [1], in terms of Fréchet Inception Distance (FID) [13], and outperform even the best state-of-the-art model by up to 29 points.”