Entity Abstraction in Visual Model-Based Reinforcement Learning

By Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Josh Tenenbaum, Sergey Levine. 2019

208 citations recorded by the source corpus.

This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses them to predict and plan. OP3 enforces entity-abstraction -- symmetric processing of each entity representation with the same locally-scoped function -- which enables it to scale to model different numbers and configurations of objects from those in training. Our approach to solving the key technical challenge of grounding these entity representations to actual objects in the environment is to frame this variable binding problem as an inference problem, and we develop an interactive inference algorithm that uses temporal continuity and interactive feedback to bind information about object properties to the entity variables. On block-stacking tasks, OP3 generalizes to novel block configurations and more objects than observed during training, outperforming an oracle model that assumes access to object supervision and achieving two to three times better accuracy than a state-of-the-art video prediction model that does not exhibit entity abstraction.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “, 2017) and Josh Tenenbaum’s group (Mao et al., 2019; Veerapaneni et al., 2019).”
Object-Centric Learning with Slot Attention (2020, background): “[68] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine.”
Neural Production Systems (2021, unknown): “We show that NPS outperforms state-of-the-art GNN-based architectures such as C-SWM Kipf et al. (2019) and OP3 Veerapaneni et al. (2019) while learning world-models.”
Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning (2020, unknown): “More recently, this inductive bias served as a source of inspiration for several recent advancements in deep learning, such as Kulkarni et al. (2016), van Steenkiste et al. (2018), Kulkarni et al. (2019), Veerapaneni et al. (2019), and Lin et al. (2020).”
Conditional Object-Centric Learning from Video (2021, methodology): “Most closely related to our work are sequential object-centric models for videos and dynamic environments, such as OP3 (Veerapaneni et al., 2020), R-SQAIR (Stanić & Schmidhuber, 2019), ViMON (Weis et al., 2020), and SCALOR (Jiang et al., 2020), which learn an internal motion model for each object.”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “Examples of such models are ones based on graph neural networks [Battaglia et al., 2016, Van Steenkiste et al., 2018, Kipf et al., 2019, Veerapaneni et al., 2020] and modular models [Ke et al., 2020, Goyal et al., 2019, Mittal et al., 2020, Goyal et al., 2020].”
Hierarchical Relational Inference (2020, background): “2019) and of dynamically inferring relationships between objects (van Steenkiste et al. 2018; Kipf et al. 2018; Goyal et al. 2019; Veerapaneni et al. 2019), reasoning about the dynamics and interactions of complex objects remains difficult without incorporating additional structure.”
Unsupervised part representation by Flow Capsules (2020, background): “Most recently, (Bear et al., 2020; Veerapaneni et al., 2020) extend such networks to incorporate motion, but still rely on iterative refinement.”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, background): “More recently, object-centric approaches have also been leveraged to acquire better video representations for future frame prediction [43] or model-based reinforcement learning [37].”
Learning Physical Graph Representations from Visual Scenes (2020, methodology): “The difference was less pronounced when training on Playroom , where MONet, IODINE, and OP3 peaked after 38, 83, and 24 epochs – compared to 1.8 and 1.0 for PSGNetS and PSGNetM; however, the poor performance of the baselines on this dataset (<0.30 test set Recall ) makes their learning efﬁciency a l”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “In recent work, several methods have been proposed to learn unsupervised representations of images that aim to facilitate agents in this way (Veerapaneni et al., 2019; Janner et al., 2019).”
Generative Neurosymbolic Machines (2020, background): “Scene-mixture models [16, 41, 6, 15, 12, 44, 32] try to partition the image into several layers of images, potentially one per object, and reconstruct the full image as a pixel-wise mixture of these layered images.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “, location and velocity) based on learned visual and dynamic priors [15, 10, 69, 68, 41, 52, 53, 32, 31].”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, methodology): “…models using pixel-level convolutional networks [31, 32, 33, 34, 29, 35] and scene-based learning [14, 36, 37, 38] to learning of physical representations of the intuitive physics [6, 2, 39] behind the interactions to make accurate prediction of the outcomes of physical interactions among objects.”
Better Set Representations For Relational Reasoning (2020, methodology): “These methods effectively decompose scenes without explicit supervision, but they often require sophisticated architectures speciﬁc to object detection and segmentation that are difﬁcult to reuse [6, 18].”