Learning to Poke by Poking: Experiential Learning of Intuitive Physics

By Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, Sergey Levine. 2016

613 citations recorded by the source corpus.

We investigate an experiential learning paradigm for acquiring an internal model of intuitive physics. Our model is evaluated on a real-world robotic manipulation task that requires displacing objects to target locations by poking. The robot gathered over 400 hours of experience by executing more than 100K pokes on different objects. We propose a novel approach based on deep neural networks for modeling the dynamics of robot's interactions directly from images, by jointly estimating forward and inverse models of dynamics. The inverse model objective provides supervision to construct informative visual features, which the forward model can then predict and in turn regularize the feature space for the inverse model. The interplay between these two objectives creates useful, accurate models that can then be used for multi-step decision making. This formulation has the additional benefit that it is possible to learn forward models in an abstract feature space and thus alleviate the need of predicting pixels. Our experiments show that this joint modeling approach outperforms alternative methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Learning from interaction and intuitive physics (Agrawal et al. 2016) can also be encoded as priors when exploring the world (Byravan et al.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “One particularly important domain for self-supervised learning is robotics, where representations can be learned by interacting with the environment [2, 53, 52, 60], and where success greatly depends on making effective use of unlabeled data.”
Multi-task Self-Supervised Visual Learning (2017, background): “In the last few years, a considerable number of such tasks have been proposed [1, 2, 6, 7, 8, 17, 20, 21, 23, 25, 26, 27, 28, 29, 31, 39, 40, 42, 43, 46, 47], such as asking a neural network to colorize grayscale images, ﬁll in image holes, solve jigsaw puzzles made from image patches, or predict…”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Some modeling choices include using pixels [28, 100], graphs [10], euclidean distance using sensors [2], or studying cycle consistency across time Epstein et al.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “The robotics community focuses on video prediction models for planning (Agrawal et al., 2016; Finn & Levine, 2017; Ebert et al., 2018) that deal with the visual complexity of the real world and solve tasks with a simple gripper, such as grasping or pushing objects.”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Agrawal et al. (2016) propose to learn a forward and inverse dynamics model from pixels with applications to robotics.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “…reasoning (Lerer et al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Battaglia et al., 2016; Chang et al., 2017; Agrawal et al., 2016; Finn et al., 2016; Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et…”
Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
Flexible Neural Representation for Physics Prediction (2018, background): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “Purely unsupervised approaches for state-to-state prediction (Agrawal et al., 2016; Lerer et al., 2016; Michalski et al., 2014; Sutskever et al., 2009) that use raw visual inputs as state-descriptions have yet to rival these capabilities.”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, background): “These systems include previously studied gravitational and billiards systems [3, 1] with the added challenge of natural image backgrounds.”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “, 2015) formulates the problem as inference over the parameters of a symbolic physics engine, while the bottom-up approach (Agrawal et al., 2016; Fragkiadaki et al., 2015; Lerer et al., 2016; Li et al., 2016; Mottaghi et al., 2015; 2016; Sutskever et al., 2009) learns to directly map physical observ”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “ation, the controller can be easily trained to converge based on back-propagation algorithms. Long-term recurrent features helped to keep the state information from previous observations effectively. Agrawal et al. (2016) used a deep neural network to predict the dynamics model of robotic arm operat”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “…in parametric function approximators, such as NNs, suggest that such methods are worth revisiting [Baranes and Oudeyer, 2013, Fu et al., 2016, Punjani and Abbeel, 2015, Lenz et al., 2015, Agrawal et al., 2016, Gal et al., 2016, Depeweg et al., 2016, Williams et al., 2017, Nagabandi et al., 2017].”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “Our approach of jointly training forward and inverse models for learning a feature space has similarities to (Agrawal et al., 2016; Jordan & Rumelhart, 1992; Wolpert et al., 1995), but these works use the learned models of dynamics for planning a sequence of actions instead of exploration.”
Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “This perspective also allows us to draw a connection between meta learning and model based active learning, where a model of the environment is used to guide an active process for collecting data to improve that model [Assael et al., 2015, Agrawal et al., 2016].”
The limits and potentials of deep learning for robotics (2018, background): “Different variants of deep learning have been shown to successfully learn predictive physics models and robot control policies in a purely data driven way [2], [17], [51], [127].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “Further, we also study how we can extend vision-based inverse models [10, 27, 6, 11] for generalizable robot-agnostic control.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Several prior works have also sought to learn inverse models that map from pairs of observations to actions, which can then be used greedily to carry out short-horizon tasks [20], [21].”