Deep spatial autoencoders for visuomotor learning

By Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, Pieter Abbeel. 2015

598 citations recorded by the source corpus.

Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, background): “Though most of such works do not use image observations, several recent works have incorporated images into real-world (Finn et al., 2016; Finn & Levine, 2017; Babaeizadeh et al., 2017a; Ebert et al., 2017; Piergiovanni et al., 2018; Paxton et al., 2019; Rybkin et al., 2018; Ebert et al., 2018) and…”
Recurrent World Models Facilitate Policy Evolution (2018, background): “Learning a model of the dynamics from a compressed latent space enable RL algorithms to be much more data-efﬁcient [15, 101].”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, methodology): “This is an alternative approach to selecting a state representation from a set ([7], [8]), or creating an autoencoder to compress information into a lower dimensional state ([9], [10]), [11].”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Unsupervised learning of object keypoints has not been widely explored in the control literature, with the notable exception of [5].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Recent works have made progress in this regard (Finn et al., 2016; Ha and Schmidhuber, 2018; Kaiser et al., 2019; Kurutach et al., 2018; Zhang et al., 2018), but model-free alternatives are still hard to beat in terms of asymptotic performance.”
World Models (2018, methodology): “Learning a model of the dynamics from a compressed latent space enable RL algorithms to be much more dataefficient (Finn et al., 2015; Watter et al., 2015; Finn, 2017).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Model-based DRL methods are also utilized by Finn et al. (2016) and Tzeng et al. (2015), learning useful state representations for generating successful control policies.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Methods based on local models, such as guided policy search algorithms [Levine et al., 2016, Finn et al., 2016, Chebotar et al., 2017], can efficiently train NN policies, but use time-varying linear models, which only locally model the system dynamics.”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “These feature extractors terminate with a spatial-softmax layer [13] that infer low dimensional keypoint representations from spatial feature representations.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “Several algorithms have been proposed for simple, synthetic images [13] and video game environments [14], [15], [16], but have not been evaluated on generalization or in the real world, while other work has also studied model-based RL for individual robotic skills [17], [18], [19].”
Deep visual foresight for planning robot motion (2016, methodology): “Other works have used model-based approaches with a learned low-dimensional embedding of images [28], [29], [30], [31], defining the objective using an image of the goal; however, such methods have been demonstrated either on simple synthetic images, or without generalization to new objects.”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “The keypoint detector is a deep convolutional network followed by a spatial softmax layer [6].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, methodology): “Sometimes polices are learned with a frozen encoder [2], [9], [10], other times in conjunction with end-to-end training [13].”
One-Shot Visual Imitation Learning via Meta-Learning (2017, methodology): “The ﬁnal convolutional layer is transformed into spatial feature points using a spatial soft-argmax [28, 29] and concatenated with the robot’s conﬁguration.”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, methodology): “A number of recent methods have also been proposed for learning representations for vision-based control using autoencoders to transform visual inputs into a low-dimensional state space (Lange et al., 2012; Finn et al., 2015; Lange & Riedmiller, 2010; Kietzmann & Riedmiller, 2009).”
Learning dexterous in-hand manipulation (2018, methodology): “Camera images are passed through a convolutional feature stack that consists of two convolutional layers, max-pooling, 4 ResNet blocks [24], and spatial softmax (SSM) [19] layers with shared weights between the feature stacks for each camera.”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, methodology): “To incorporate vision into our experiments, we used unsupervised learning to acquire a vision-based state representation, following prior work (Finn et al., 2015).”