Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions

By Sjoerd van Steenkiste, Michael Chang, Klaus Greff, J. Urgen Schmidhuber. 2018

302 citations recorded by the source corpus.

Common-sense physical reasoning is an essential ingredient for any intelligent agent operating in the real-world. For example, it can be used to simulate the environment, or to infer the state of parts of the world that are currently unobserved. In order to match real-world conditions this causal knowledge must be learned without access to supervised data. To address this problem we present a novel method that learns to discover objects and model their physical interactions from raw visual images in a purely \emph{unsupervised} fashion. It incorporates prior knowledge about the compositional nature of human perception to factor interactions between object-pairs and learn efficiently. On videos of bouncing balls we show the superior modelling capabilities of our method compared to other unsupervised neural approaches that do not incorporate such prior knowledge. We demonstrate its ability to handle occlusion and show that it can extrapolate learned knowledge to scenes with different numbers of objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[39] Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber.”
Toward Causal Representation Learning (2021, background): “Note that the problem of object-centric representation learning [11], [40], [84], [87], [88], [138], [155], [160], [255], [262] can also be considered a special case of disentangled factorization as discussed here.”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Alternatively, multiple different relations could be learned by also conditioning the message on the receiving object representation when using attention e.g. as in R-NEM (van Steenkiste et al., 2018).”
Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems (2020, methodology): “For ex. authors in [Van Steenkiste et al., 2018] claim that the LSTM is not able to track the ball through the occlusion (i.e curtain task) but the LSTM baseline used in this work is able to track the ball through the occlusion.”
Towards causal generative scene models via competition of experts (2020, methodology): “Clustering & spatial mixture models One line of work (Greff et al., 2016; 2017; Van Steenkiste et al., 2018) approaches the perceptual grouping task of decomposing scenes into components by viewing separate regions r t as clusters.”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, methodology): “Works based on Neural Expectation Maximization (Van Steenkiste et al., 2018; Greff et al., 2017) do achieve unsupervised object detection but do not explicitly model the presence, appearance, and location of objects.”
Recurrent Independent Mechanisms (2019, methodology): “…scenes, and motion-capture data, and multiagent systems (Scarselli et al., 2008; Bronstein et al., 2017; Watters et al., 2017; Raposo et al., 2017; Santoro et al., 2017; Gilmer et al., 2017; Van Steenkiste et al., 2018; Kipf et al., 2018; Battaglia et al., 2018; Tacchetti et al., 2018).”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Finally, while this paper focuses on evaluating the benefits of disentangled features, these are complementary to recent work that focuses on the unsupervised “disentangling” of images into compositional primitives given by object-like representations [17, 23, 24, 22, 60, 74, 75].”
Structured agents for physical construction (2019, methodology): “Within the domain of physical reasoning, graph neural networks have been used as forward models for predicting future states and images (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; van Steenkiste et al., 2018), and can allow efﬁcient learning and rich generalization.”
Multi-Object Representation Learning with Iterative Variational Inference (2019, background): “In line with recent advances (Burgess et al., 2019; van Steenkiste et al., 2018; Eslami et al., 2016), we maintain that discovery of objects in a scene should be considered a crucial aspect of representation learning, rather than treated as a separate problem.”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “( Vondrick et al., 2016a) train a conditional CNN to predict future feature vectors, and several papers (e.g., (Lerer et al., 2016; Mottaghi et al., 2016; Wu et al., 2017; van Steenkiste et al., 2018; Fragkiadaki et al., 2016; Watters et al., 2017)) predict future object states.”
MONet: Unsupervised Scene Decomposition and Representation (2019, methodology): “…line of work spanning Tagger [Greﬀ et al., 2016], Neural Expectation Maximization [Greﬀ et al., 2017], and Relational Neural Expectation Maximization [van Steenkiste et al., 2018] makes use of iterative reﬁnement to decompose scenes into groups of pixels and assign them to individual components.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “They have also been used to learn the dynamics of physical systems (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; van Steenkiste et al., 2018; Sanchez-Gonzalez et al., 2018) and multi-agent systems (Sukhbaatar et al., 2016; Hoshen, 2017; Kipf et al., 2018), to reason about…”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “…et al., 2017 learn to decompose images into separate entities by iterative reﬁnement of spatial clusters using either learned updates or the Expectation Maximization algorithm; Ilin et al., 2017 and Steenkiste et al., 2018 extend these approaches to videos, achieving very similar results to SQAIR .”
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, background): “In the case of self-attention the query, key, and value vectors are embeddings of the input features: q=hQ(f), k=hK(f), v=hV (f) where {hQ, hK , hV } are, in the most general case, neural networks [22].”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “Next, we evaluate how DDPAE can be applied to videos involving more complex interactions between components on the Bouncing Balls dataset [3, 36].”