Learning Dexterous Manipulation Policies from Experience and Imitation

By Vikash Kumar, Abhishek Gupta, E. Todorov, Sergey Levine. 2016

88 citations recorded by the source corpus.

We explore learning-based approaches for feedback control of a dexterous five-finger hand performing non-prehensile manipulation. First, we learn local controllers that are able to perform the task starting at a predefined initial state. These controllers are constructed using trajectory optimization with respect to locally-linear time-varying models learned directly from sensor data. In some cases, we initialize the optimizer with human demonstrations collected via teleoperation in a virtual environment. We demonstrate that such controllers can perform the task robustly, both in simulation and on the physical platform, for a limited range of initial conditions around the trained starting state. We then consider two interpolation methods for generalizing to a wider range of initial conditions: deep learning, and nearest neighbors. We find that nearest neighbors achieve higher performance. Nevertheless, the neural network has its advantages: it uses only tactile and proprioceptive feedback but no visual feedback about the object (i.e. it performs the task blind) and learns a time-invariant policy. In contrast, the nearest neighbors method switches between time-varying local controllers based on the proximity of initial object states sensed via motion capture. While both generalization methods leave room for improvement, our work shows that (i) local trajectory-based controllers for complex non-prehensile manipulation tasks can be constructed from surprisingly small amounts of training data, and (ii) collections of such controllers can be interpolated to form more global controllers. Results are summarized in the supplementary video: this https URL

Subfields: Biomedical Engineering, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Cognitive Neuroscience

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “In contrast, multiple authors learn policies for dexterous in-hand manipulation directly on the robot. Hoof et al. [114] learn in-hand manipulation for a simple 3-ﬁngered gripper whereas Kumar et al. [53, 52] and Falco et al. [31] learn such policies for more complex humanoid hands. In [71], the aut”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “Prior work has demonstrated learning of simpler manipulation tasks like twirling a cylinder with a similar morphology [26] using guided policy search [28], and extended this approach to also incorporate demonstrations [23].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Reinforcement learning has been applied to a wide variety of robotic manipulation tasks, including grasping objects [19], in-hand object manipulation [30, 38, 32, 23], manipulating flu-”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “Existing use cases of VR for learning policies have so far been limited to collecting waypoints of low-dimensional robot states [43], [44].”
Learning dexterous in-hand manipulation (2018, background): “Conversely, due to the difficulty in modeling such complex systems, there has also been work in approaches that only train on a physical robot [16, 67, 29, 30].”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, background): “Gupta et al. [11] and Kumar et al. [21] use GPS for learning controllers for robotic hand models.”
Towards Generalization and Simplicity in Continuous Control (2017, background): “Recent success of RL [2, 8, 9, 10, 11] has been enabled largely due to engineering efforts such as large scale data collection [1, 2, 11] or careful systems design [8, 9] with well behaved robots.”
On Policy Learning Robust to Irreversible Events: An Application to Robotic In-Hand Manipulation (2018, methodology): “A deep learning approach combined with demonstrations is adopted in [17], to perform in-hand manipulation tasks in a simulated environment, while in [18] an in-hand non-prehensile manipulation is executed starting from human demonstration.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “Robotic RL. RL has been applied to a wide variety of robotic manipulation tasks, including grasping objects [11, 12], in-hand object manipulation [13, 14, 15, 16], pouring ﬂuids [17], door opening [18, 19], and manipulating cloth [20, 21].”
Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation (2018, methodology): “That is why more and more studies use synthetic data to train imitation learning methods in a virtual environment [6,34,16,41].”
Catch & Carry (2019, methodology): “Finally, as noted previously, there are approaches in which both matching the demonstrations and solving the task serve as rewards during training [Kumar et al. 2016; Peng et al. 2018; Merel et al. 2017; Zhu et al. 2018].”
A System for General In-Hand Object Re-Orientation (2021, methodology): “Model-based RL works learned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and used online optimal control to rotate a pen or Baoding balls on a Shadow hand.”
A Review of Tactile Information: Perception and Action Through Touch (2020, unknown): “Alternatively,tactilesensingcanalsobeusedtodirectly learnamappingfromthecontactsensorinformationtothe desiredjointvelocitiesoftheroboticﬁngers[144],[201]using data-drivenapproaches.”