MONet: Unsupervised Scene Decomposition and Representation

By Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, Alexander Lerchner. 2019

617 citations recorded by the source corpus.

The ability to decompose scenes in terms of abstract building blocks is crucial for general intelligence. Where those basic building blocks share meaningful properties, interactions and other regularities across scenes, such decompositions can simplify reasoning and facilitate imagination of novel scenarios. In particular, representing perceptual observations in terms of entities should improve data efficiency and transfer performance on a wide range of tasks. Thus we need models capable of discovering useful decompositions of scenes by identifying units with such regularities and representing them in a common format. To address this problem, we have developed the Multi-Object Network (MONet). In this model, a VAE is trained end-to-end together with a recurrent attention network -- in a purely unsupervised manner -- to provide attention masks around, and reconstructions of, regions of images. We show that this model is capable of learning to decompose and represent challenging 3D scenes into semantically meaningful components, such as objects and background elements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “See also DeepMind’s MONet (Burgess et al., 2019)and MERLIN (Wayne et al., 2018) as well the expressive generative model in (Gregor et al., 2019)] and. e.g., inverse graphics papers by Vicarious (Kansky et al., 2017; George et al., 2017) and Josh Tenenbaum’s group (Mao et al., 2019; Veerapaneni et al”
Object-Centric Learning with Slot Attention (2020, methodology): “In general, we observe that our model compares favorably against two recent state-of-the-art baselines: IODINE [16] and MONet [17].”
Towards causal generative scene models via competition of experts (2020, methodology): “In particular, we build on the recent MONET model for scene decomposition of Burgess et al. (2019).”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, background): “In scenemixture models (Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2019), a visual scene is explained by a mixture of a finite number of component images.”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Both instance and semantic segmentation has been approached using supervised [22, 25] and unsupervised learning [2, 9, 14, 10, 16, 21, 6] methods.”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “As shown in Burgess et al. (2019), the MONet model not only learns a scene decomposition but can also learn a disentangled latent space for object representations.”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “However, unlike our model, MONet does not utilize any iterative reﬁnement, which we believe will be important for scaling up to even more challenging datasets.”
Stacked Capsule Autoencoders (2019, methodology): “The image is modelled as a spatial Gaussian mixture, similarly to Greff et al., 2019; Burgess et al., 2019; Engelcke et al., 2019.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “…a mixture model, where the mixing probabilities π k = π θ ( z m 1: k ) are created via a stick-breaking process ( SBP ) adapted from Burgess et al. [6] as follows, slightly overloading the π notation, Note that this step is not essential for our model and instead we could use a softmax to…”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, methodology): “…model ranking produced by UDR correlates with task performance on two different domains: the fairness on a classiﬁcation task introduced by Locatello et al. (2018), and data efﬁciency on a clustering task for a model-based reinforcement learning agent introduced by Watters et al. (2019) (see Sec.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Recent works therefore investigate how the synthesis process can be controlled at the object-level [3,4,7,18,19,26,45,86,90].”
Object Scene Representation Transformer (2022, methodology): “We believe this to be a result of uORF’s already limited capacity, which we had to further constrain by reducing the slot dimensionality to 10 to ﬁt a 32-slot model into an Nvidia A100 GPU with 40 GB of VRAM.”
Generative Adversarial Transformers (2021, background): “Several approaches, such as [7, 22, 23, 31], perform iterative variational inference to encode scenes into multiple slots, but are mostly applied in the contexts of synthetic and oftentimes fairly rudimentary 2D settings.”
Neural Production Systems (2021, methodology): “…al., 2011; Eslami et al., 2016; Greff et al., 2016; Raposo et al., 2017; Van Steenkiste et al., 2018; Kosiorek et al., 2018; Engelcke et al., 2019; Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020a; Ahmed et al., 2020; Goyal et al., 2019; Zablotskaia et al., 2020; Rahaman et al.,…”