Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

By Michael Janner, Sergey Levine, William T. Freeman, Josh Tenenbaum, Chelsea Finn, Jiajun Wu. 2018

136 citations recorded by the source corpus.

Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Related work by Janner et al (Janner et al., 2018). pairs explicit records for individual objects with deep learning in order to make predictions and physics-based plans that far surpass a comparable pure black box deeplearning approach.”
On the Binding Problem in Artificial Neural Networks (2020, background): “MPNNs have been shown to generalize more systematically (compared to standard neural networks) on many different tasks that require relational responding in terms of objects, including common-sense physical reasoning (Chang et al., 2017; Battaglia et al., 2016; Janner et al., 2019), hierarchical phy”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “It is worth noting that similar setups were proposed previously in a less realistic setting as in (Janner et al., 2018; Bapst et al., 2019; McCarthy et al.; Akkaya et al., 2019; Fahlman, 1974; Winston, 1970; Winograd, 1972).”
Structured agents for physical construction (2019, methodology): “…an unstable stack (Hamrick et al., 2018), learning basic block-stacking by predicting shortest paths between current and goal states via a transition model (Zhang et al., 2018), and learning object representations and coarse-grained physics models for stacking blocks (Janner et al., 2019).”
The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation (2020, result): “ent works in vision-based control and intuitive physics have argued that instead of identifying the output dynamics, a neural network-based observer can observe the object-centric state of the system [8,12,9,26,23]. Since object-centric states are lower dimensional and have more identiable continuou”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, background): “These works either used thousands of expert demonstrations [42], [43] or were limited to their application on video games[19].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Other works have also explored learning hierarchical policies on top of hand-designed skills [70, 71, 72].”
PHYRE: A New Benchmark for Physical Reasoning (2019, methodology): “Early computational instantiations of this framework used probabilistic models over physical simulations [3, 61], while more recent methods use feedforward neural networks trained to make pixelwise [11, 24, 34, 60] or qualitative [12, 26, 27, 34, 61] predictions about the future, sometimes in conjuc”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “, 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “More complex hierarchical models, such as neural networks and decision trees, allow the robot to learn taskspecific features that provide better generalization [235, 334, 117, 174, 321, 110, 230, 353, 431].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “, location and velocity) based on learned visual and dynamic priors [15, 10, 69, 68, 41, 52, 53, 32, 31].”
Contrastive Learning of Structured World Models (2019, methodology): “While most methods in this area require some form of human annotation for the extraction of objects or relations, several recent works study the problem of object discovery from visual data in a completely unsupervised or self-supervised manner (Eslami et al., 2016; Greff et al., 2017; Nash et al., ”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, methodology): “Janner et al. [35] showed that an object-centric model with access to ground truth object segmentations can solve these tasks with about 76% accuracy.”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “Such learned approximations of object interactions have been successfully employed in object manipulation [15] and tracking [9, 21].”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, methodology): “This approach is often used when applying model predictive control (Feldbaum, 1960; C. Xie, Patil, Moldovan, Levine, & Abbeel, 2016; Fu, Levine, & Abbeel, 2016; Janner et al., 2018; Deisenroth & Rasmussen, 2011), also often called system identiﬁcation (Chiuso & Pillonetto, 2019).”
Visual Grounding of Learned Physical Models (2020, unknown): “Other researchers have shown better performance with intermediate representations like instance masks (Fragkiadaki et al., 2016; Watters et al., 2017; Janner et al., 2019), ob-Visual ject keypoints (Minderer et al., 2019), or dense visual descriptors (Xu et al., 2019).”