Neural Expectation Maximization

By Klaus Greff, Sjoerd van Steenkiste, J. Urgen Schmidhuber. 2017

309 citations recorded by the source corpus.

Many real world tasks such as reasoning and physical interaction require identification and manipulation of conceptual entities. A first step towards solving these tasks is the automated discovery of distributed symbol-like representations. In this paper, we explicitly formalize this problem as inference in a spatial mixture model where each component is parametrized by a neural network. Based on the Expectation Maximization framework we then derive a differentiable clustering method that simultaneously learns how to group and represent individual entities. We evaluate our method on the (sequential) perceptual grouping task and find that it is able to accurately recover the constituent objects. We demonstrate that the learned representations are useful for next-step prediction.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[25] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2016), generalized expectation-maximization (Greff et al., 2017), iterative variational inference (Greff et al.”
Towards causal generative scene models via competition of experts (2020, methodology): “Clustering & spatial mixture models One line of work (Greff et al., 2016; 2017; Van Steenkiste et al., 2018) approaches the perceptual grouping task of decomposing scenes into components by viewing separate regions r t as clusters.”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, methodology): “In scenemixture models (Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2019), a visual scene is explained by a mixture of a finite number of component images.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Finally, while this paper focuses on evaluating the benefits of disentangled features, these are complementary to recent work that focuses on the unsupervised “disentangling” of images into compositional primitives given by object-like representations [17, 23, 24, 22, 60, 74, 75].”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “We approach the problem from a spatial mixture model perspective (Greff et al., 2017) and use amortized iterative reﬁnement (Marino et al., 2018b) of latent object representations within a variational framework (Rezende et al., 2014; Kingma & Welling, 2013).”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “Another line of work spanning Tagger [Greﬀ et al., 2016], Neural Expectation Maximization [Greﬀ et al., 2017], and Relational Neural Expectation Maximization [van Steenkiste et al., 2018] makes use of iterative reﬁnement to decompose scenes into groups of pixels and assign them to individual…”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “This work continues the thread of Greff, Steenkiste, et al., 2017, Steenkiste et al., 2018 and, together with Hsieh et al., 2018, presents unsupervised object detection & tracking with learnable likelihoods by the means of generative modelling of objects.”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, methodology): “At its core is Neural Expectation Maximization (N-EM; Greff et al., 2017), a method that allows for the discovery of compositional object-representations, yet is unable to model interactions between objects.”
Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks (2019, background): “Recent work on Neural Expectation Maximization (Greff, van Steenkiste, and Schmidhuber 2017; van Steenkiste et al. 2018) aims to achieve something similar to AIR, but has comparable scaling issues.”
The limits and potentials of deep learning for robotics (2018, background): “…grounded in real world experience, are well suited for closed-loop control, and can be learned and Recent approaches to end-to-end training of deep networks forgo the use of explicit physics models, learning predictive models and controls from raw experiences [9], [16], [18], [36], [129], [132].”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, background): “Existing solutions, however, either require expensive inference procedures (Greff et al. [7, 15, 16]) or lack a proper probabilistic formulation (Burgess et al. [6]).”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “It has been used in many previous works [6, 12, 15, 24, 31].”
One Big Net For Everything (2018, background): “…of q special output units and train them through unsupervised learning [62] to produce for t ≤ t T a vector code ( t ) ∈ R q that represents an ideal factorial code [2] of the observed history so far, or that encodes the data in related ways generally considered useful, e.g., [23, 28, 81, 68, 21].”
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning (2018, background): “There has also been much attention on inferring object-factorized, or otherwise disentangled, representations of images (Eslami et al., 2016; Greff et al., 2017; van Steenkiste et al., 2018).”
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, methodology): “For unsupervised learning, these ideas have been mainly used to model the data distribution [8] and have not been applied to unsupervised training of regression or classification problems.”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, background): “Recent approaches can be broadly categorized into spatial mixture models (Greff et al., 2017; 2019; Burgess et al., 2019), sequential attention models (Eslami et al.”
Generative Neurosymbolic Machines (2020, background): “Scene-mixture models [16, 41, 6, 15, 12, 44, 32] try to partition the image into several layers of images, potentially one per object, and reconstruct the full image as a pixel-wise mixture of these layered images.”
Conditional Object-Centric Learning from Video (2021, background): “…representation learning There is a rich literature on learning object representations from static scenes (Greff et al., 2016; Eslami et al., 2016; Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2020; Crawford & Pineau, 2019; Lin et al., 2020; Locatello et al., 2020; Du et al.,…”