Multi-Object Representation Learning with Iterative Variational Inference

By Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nicholas Watters, Christopher P. Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, Alexander Lerchner. 2019

583 citations recorded by the source corpus.

Human perception is structured around objects which form the basis for our higher-level cognition and impressive systematic generalization abilities. Yet most work on representation learning focuses on feature learning without even considering multiple objects, or treats segmentation as an (often supervised) preprocessing step. Instead, we argue for the importance of learning to segment and represent objects jointly. We demonstrate that, starting from the simple assumption that a scene is composed of multiple entities, it is possible to learn to segment images into interpretable objects with disentangled representations. Our method learns -- without supervision -- to inpaint occluded parts, and extrapolates to scenes with more objects and to unseen objects with novel feature combinations. We also show that, due to the use of iterative variational inference, our system is able to learn multi-modal posteriors for ambiguous inputs and extends naturally to sequences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “Decoder Each slot is decoded individually with the help of a spatial broadcast decoder [28], as used in IODINE [16]: slot representations are broadcasted onto a 2D grid (per slot) and augmented with position embeddings.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “In preliminary experiments on the Tetrominoes object discovery task from Greff et al. (2019), we only inconsistently reached above 98% on the ARI evaluation metric, unable to match the 5-run average of 99.5% ARI for Slot Attention (Locatello et al., 2020) and 99.2% ARI for IODINE (Greff et al., 2019”
Toward Causal Representation Learning (2021, background): “Note that the problem of object-centric representation learning [11], [40], [84], [87], [88], [138], [155], [160], [255], [262] can also be considered a special case of disentangled factorization as discussed here.”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2019), or by using neural mixture models (Greff et al., 2019), or an adversarial formulation (Yang et al.”
Towards causal generative scene models via competition of experts (2020, methodology): “While the scenes studied here and in the recent works of Burgess et al. (2019); Greff et al. (2019); Engelcke et al. (2019) are still in stark contrast to the impressive results that holistic generative models are able to achieve, we believe it is the right time to revisit the unsupervised scene…”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Both instance and semantic segmentation has been approached using supervised [22, 25] and unsupervised learning [2, 9, 14, 10, 16, 21, 6] methods.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “representation that captures this product space is expected to help in generalizing systematically in this regard [18, 22, 59].”
Stacked Capsule Autoencoders (2019, methodology): “The image is modelled as a spatial Gaussian mixture, similarly to Greff et al., 2019; Burgess et al., 2019; Engelcke et al., 2019.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “…probabilities π k = π θ ( z m 1: k ) are created via a stick-breaking process ( SBP ) adapted from Burgess et al. [6] as follows, slightly overloading the π notation, Note that this step is not essential for our model and instead we could use a softmax to normalise masks as in Greff et al. [7].”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, methodology): “Results in Table 3 show that ROOTS outperforms IODINE on both foreground segmentation (ARI-NoBg) and full image segmentation (ARI).”
Generative Adversarial Transformers (2021, background): “Several variational approaches [7, 22, 23, 31] perform iterative inference to encode scenes into multiple slots, but are mostly applied in the contexts of synthetic and oftentimes fairly rudimentary 2D settings.”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “More recently, there has been renewed interest in the problem of unsupervised grouping [35, 51, 34, 25, 52].”
Learning To Classify Images Without Labels (2020, methodology): “importantly, since the cluster learning depends on the network initialization, it is likely to latch onto low-level features, such as color, which is unwanted for the objective of semantic clustering [16]. To prevent the network from focusing on the low-level information, certain approaches [24,21] ”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, methodology): “OP3 builds upon recently proposed ideas in grounding entity representations via inference on a symmetrically factorized generative model of static [8, 17, 42] and dynamic [43] scenes, whose advantage over other methods for grounding [5, 44–47] is the ability to reﬁne the grounding with new…”
Learning Physical Graph Representations from Visual Scenes (2020, methodology): “Recently, machine learning researchers have developed deep generative models for unsupervised segmentation by capturing relations between scene components [10, 4, 17, 9], with varying degrees of success and generality.”
Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views (2021, methodology): “However, this comes at the cost of decreased representational power (good object representation requires good object segmentation [9]) and a reliance on annotated data.”