Learning contact-rich manipulation skills with guided policy search

By Sergey Levine, Nolan Wagener, Pieter Abbeel. 2015

358 citations recorded by the source corpus.

Autonomous learning of object manipulation skills can enable robots to acquire rich behavioral repertoires that scale to the variety of objects found in the real world. However, current motion skill learning methods typically restrict the behavior to a compact, low-dimensional representation, limiting its expressiveness and generality. In this paper, we extend a recently developed policy search method [1] and use it to learn a range of dynamic manipulation behaviors with highly general policy representations, without using known models or example demonstrations. Our approach learns a set of trajectories for the desired motion skill by using iteratively refitted time-varying linear models, and then unifies these trajectories into a single control policy that can generalize to new situations. To enable this method to run on a real robot, we introduce several improvements that reduce the sample count and automate parameter selection. We show that our method can acquire fast, fluent behaviors after only minutes of interaction time, and can learn robust controllers for complex tasks, including putting together a toy airplane, stacking tight-fitting lego blocks, placing wooden rings onto tight-fitting pegs, inserting a shoe tree into a shoe, and screwing bottle caps onto bottles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “…set for this supervised learning procedure can be constructed from example demonstrations [20], trajectory optimization under known dynamics [21, 22, 28], and trajectory-centric reinforcement learning methods that operate under unknown dynamics [19, 23], which is the approach taken in this work.”
Learning to Optimize (2016, methodology): “We use an off-the-shelf reinforcement learning algorithm known as guided policy search [17], which has demonstrated success in a variety of robotic control settings [18, 10, 19, 12].”
Solving Rubik's Cube with a Robot Hand (2019, background): “Guided policy search [56, 58] learns simple local policies directly on the robot and distills them into a global policy represented by a neural network.”
Sim-to-Real Robot Learning from Pixels with Progressive Nets (2016, background): “remains difﬁcult, progress has been made with directly learning neural network control policies on a real robot, both from low-dimensional representations of the state and from visual input (e.g. [9],[11]). While the results are impressive, to achieve sufﬁcient data efﬁciency these works currently r”
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours (2015, background): “A planar grasp is one where the grasp conﬁguration is along and perpendicular to the workspace.”
Benchmarking Model-Based Reinforcement Learning (2019, background): “In this paper, we use the MD-GPS variant [35].”
Deep spatial autoencoders for visuomotor learning (2015, background): “A full description of guided policy search is outside the scope of this paper, and we refer the reader to previous work for details [7], [8], [23].”
Learning dexterous in-hand manipulation (2018, background): “Guided policy search [31, 33] learns simple local policies directly on the robot and distills them into a global policy represented by a neural network.”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, methodology): “The step size (cid:15) can be chosen adaptively based on the degree to which the linear-Gaussian dynamics are successful in predicting the current cost (Levine et al., 2015).”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “Guided Policy Search (GPS) [8] represents one of the few algorithms capable of training policies directly on a real robot.”
The Curious Robot: Learning Visual Representations via Physical Interactions (2016, background): “There has been a recent trend to use deep networks in robotics systems for grasp regression [25,35,36] or learning policies for a variety of tasks [37,38].”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, methodology): “We use a variant of the reward function suggested by [15]: rσ = wσ 2 + v log(σ(2) + a), where σ is the angle of the pole.”
Learning dexterous manipulation for a soft robotic hand from human demonstrations (2016, neutral): “Policy search methods have succeeded in training neural network controllers to solve contact-rich peg-in-holelike tasks [17] based on positional or visual feedback [18].”
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors (2015, methodology): “learning method [21], though in practice this data could have also been generated using prior experience from the method presented in this paper, human demonstrations, or any other method that provides good coverage of interesting states (e.”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, neutral): “stage trains an initial controller using a combination of linear-Gaussian controllers and guided policy search [3].”
Learning Dexterous Manipulation Policies from Experience and Imitation (2016, methodology): “These types of controllers have been employed previously for controlling lower-dimensional robotic arms (Lioutikov, Paraschos, Neumann and Peters 2014; Levine, Wagener and Abbeel 2015b).”
Pick and Place Without Geometric Object Models (2017, background): “Since this survey, deep RL has become prominent in robotic manipulation [4], [5], [21], [22], [23].”