Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

By Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, Vikash Kumar. 2018

238 citations recorded by the source corpus.

Dexterous multi-fingered robotic hands can perform a wide range of manipulation skills, making them an appealing component for general-purpose robotic manipulators. However, such hands pose a major challenge for autonomous control, due to the high dimensionality of their configuration space and complex intermittent contact interactions. In this work, we propose deep reinforcement learning (deep RL) as a scalable solution for learning complex, contact rich behaviors with multi-fingered hands. Deep RL provides an end-to-end approach to directly map sensor readings to actions, without the need for task specific models or policy classes. We show that contact-rich manipulation behavior with multi-fingered hands can be learned by directly training with model-free deep RL algorithms in the real world, with minimal additional assumption and without the aid of simulation. We learn to perform a variety of tasks on two different low-cost hardware platforms entirely from scratch, and further study how the learning can be accelerated by using a small number of human demonstrations. Our experiments demonstrate that complex multi-fingered manipulation skills can be learned in the real world in about 4-7 hours for most tasks, and that demonstrations can decrease this to 2-3 hours, indicating that direct deep RL training in the real world is a viable and practical alternative to simulation and model-based control. https:// sites.google.com/view/deeprl-handmanipulation

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “One class of algorithms initializes the policy via behavioral cloning from demonstrations, and then fine-tunes with reinforcement learning [33, 15, 40, 18, 35, 10, 50].”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “The hand is based on the “dclaw” hand, discussed by (Zhu et al., 2018).”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “Valve Turning: This starter task for looking into manipulation challenges uses a 9-DoF hand (D’Claw) [5] to turn a 1-DoF valve to arbitrary target locations.”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “Certain systems such as dexterous hands and tensegrity robots not only exhibit complex state spaces but also complex action spaces (Zhu et al., 2018; Andrychowicz et al., 2018; Zhang et al., 2017), and learning simpler action representations that can potentially capture high-level behavior, such as…”
A Framework for Efficient Robotic Manipulation (2020, methodology): “Though there exist prior frameworks for efficient position controlled robotic manipulation [49], they still require”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, background): “Deep reinforcement learning has also been applied to learn a variety of dexterous manipulation skills using low-cost robotic hands [27].”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, methodology): “In our next set of manipulation experiments, we use the 9-DoF 3-fingered DClaw robot to rotate a valve [35] from various starting positions to various desired goal locations, where the goal is illustrated as a green dot in the image.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “…policy gradient algorithms have been used to learn tasks such as peg insertion (Lee et al. 2019), targeted throwing Ghadirzadeh et al. (2017), and dexterous manipulation (Zhu et al. 2019) directly on real-world hardware, and can be further accelerated with example demonstrations (Zhu et al. 2019).”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “The DClaw and DKitty tasks use physically accurate models of robots [53, 54].”
Goal-conditioned Imitation Learning (2019, background): “Nevertheless, these successes are shyly echoed in real world robotics [6, 7].”
Efficient Exploration via State Marginal Matching (2019, background): “It also incorporates reward shaping terms that place lower probability mass on states with high joint velocity and on states with joint positions that deviate far from the initial position (see (Zhu et al., 2019)).”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “, Rubik’s cube [1]) or by incorporating explicit human demonstrations [46], [12], [39], [15], [57], [47], [38].”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, background): “For instance, RL has been gradually mastering skills such as robot arm/hand manipulation on an object [2, 48, 36, 22] and navigation to a target destination [18, 39].”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Recent advances have greatly improved data efﬁciency, scalability, and stability of these algorithms in a variety of domains (Rennie et al., 2017; Zoph & Le, 2017; Espeholt et al., 2018; Ganin et al., 2018; Zhu et al., 2018).”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “Prior works have considered instrumenting the environment with additional sensors to evaluate rewards (Gu et al., 2017; Chebotar et al., 2017; Zhu et al., 2019), which is a highly manual process, using demonstrations, which require manual effort to collect (Vecerik et al.”