The Curious Robot: Learning Visual Representations via Physical Interactions

By Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong‐Lae Park, Abhinav Kumar Gupta. 2016

195 citations recorded by the source corpus.

What is the right supervisory signal to train visual representations? Current approaches in computer vision use category labels from datasets such as ImageNet to train ConvNets. However, in case of biological agents, visual representation learning does not require millions of semantic labels. We argue that biological agents use physical interactions with the world to learn visual representations unlike current vision systems which just use passive observations (images and videos downloaded from web). For example, babies push objects, poke them, put them in their mouth and throw them to learn representations. Towards this goal, we build one of the first systems on a Baxter platform that pushes, pokes, grasps and observes objects in a tabletop environment. It uses four different types of physical interactions to collect more than 130K datapoints, with each datapoint providing supervision to a shared ConvNet architecture allowing us to learn visual representations. We show the quality of learned representations by observing neuron activations and performing nearest neighbor retrieval on this learned representation. Quantitatively, we evaluate our learned ConvNet on image classification tasks and show improvements compared to learning without external data. Finally, on the task of instance retrieval, our network outperforms the ImageNet network on recall@1 by 3%

Subfields: Artificial Intelligence, Biophysics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, background): “In order to overcome the bottleneck, there have been recent efforts on visual representation learning using web-supervision [2, 5, 6, 9, 21, 23, 24, 27] or unsupervised [10, 11, 31, 32, 34, 42, 43] paradigms.”
Multi-task Self-Supervised Visual Learning (2017, background): “Tasks that use auxiliary information such as multi-modal information beyond pixels include: predicting sound given videos [26], predicting camera motion given two images of the same scene [1, 17, 44], or predicting what robotic motion caused a change in a scene [2, 29, 30, 31, 32].”
Representation Learning by Learning to Count (2017, background): “Other recent works use as surrogate labels ego-motion [1, 17], temporal ordering in video [27, 32], sound [31], and physical interaction [34].”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “Some examples of these auxiliary tasks include predicting: the relative location of two patches [2], ego-motion in videos [15,16], physical signals [17,18,19].”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, methodology): “Our approach can also be compared with [6], which uses several kinds of robot interactions to train a siamese neural network.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “A number of recent works have studied self-supervised robotic learning, where large-scale unattended data collection is used to learn individual skills such as grasping [22], [23], [24], [25], push-grasp synergies [26], or obstacle avoidance [27], [28].”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, background): “This has been followed by attempts to scale even further using multiple robots [3], multiple tasks [19], adversarial learning [20] and curriculum learning [21].”
Combining self-supervised learning and imitation for vision-based rope manipulation (2017, methodology): “Following the works of [21], [32], we use deep convolutional neural networks to learn the inverse model.”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “[25] learn visual representations by pushing, grasping and poking objects but do not consider using the learnt models to achieve desired goals.”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “Several recent papers address learning intuitive physics by using physical interactions (robotics) [20, 1].”
Temporal Relational Reasoning in Videos (2017, methodology): “Physical interaction and observations are also used to train deep neural networks [15].”
Tracking Emerges by Colorizing Videos (2018, background): “Other approaches include interaction with an environment to learn visual features [17–19], which is useful for applications in robotics.”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, positive): “Pinto et al. (2016); Pinto & Gupta (2016); Agrawal et al. (2016); Levine et al. (2016) are exceptions to this because they learn their models using a sequential greedy data collection bootstrapping strategy.”
Physics 101: Learning Physical Object Properties from Unlabeled Videos (2016, background): “Recently, researchers started to tackle concrete scenarios for understanding physics from vision [11, 15, 18, 19, 23, 24, 25, 27], some building on recent advances in deep learning.”
Visual Semantic Planning Using Deep Successor Representations (2017, background): “Recent work has enabled faster learning and produced more robust visual representations [1, 32, 39] through interaction.”
Learning from Noisy Large-Scale Datasets with Minimal Supervision (2017, background): “ing [17], self-supervised learning [9, 24, 25, 31] and learning from noisy annotations [6, 23].”
Supervision via competition: Robot adversaries for learning tasks (2016, methodology): “Data-driven approaches in robotics have demonstrated significant improvements in recent years [1]–[4], [11].”
State Representation Learning for Control: An Overview (2018, methodology): “For example, visual assessment of the representation’s quality can be done using a Nearest-Neighbors approach as in [Sermanet et al., 2017, Pinto et al., 2016].”
Learning Awareness Models (2018, background): “Several authors have implemented intrinsic motivation, or curiosity based objectives in visual space, through predicting interactions with objects (Pinto et al., 2016), or through predicting summary statistics of the future (Venkatraman et al., 2017; Downey et al., 2017).”