Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

By Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, E. Todorov, Sergey Levine. 2017

1,392 citations recorded by the source corpus.

Dexterous multi-fingered hands are extremely versatile and provide a generic way to perform multiple tasks in human-centric environments. However, effectively controlling them remains challenging due to their high dimensionality and large number of potential contacts. Deep reinforcement learning (DRL) provides a model-agnostic approach to control complex dynamical systems, but has not been shown to scale to high-dimensional dexterous manipulation. Furthermore, deployment of DRL on physical systems remains challenging due to sample inefficiency. Thus, the success of DRL in robotics has thus far been limited to simpler manipulators and tasks. In this work, we show that model-free DRL with natural policy gradients can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments. Furthermore, with the use of a small number of human demonstrations, the sample complexity can be significantly reduced, and enable learning within the equivalent of a few hours of robot experience. We demonstrate successful policies for multiple complex tasks: object relocation, in-hand manipulation, tool use, and door opening.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “A popular class of on-policy algorithms is policy gradient methods (Williams, 1992; Sutton et al., 2000), which have been shown to be effective for a diverse array of complex tasks (Heess et al., 2017; Pathak et al., 2017; Peng et al., 2018; Rajeswaran et al., 2018).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “The more complex Adroit [53] tasks (shown on the right) in D4RL require controlling a 24-DoF robotic hand, using limited data from human demonstrations.”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “We use the challenging Ant Maze D4RL domains (Fu et al., 2020), as well as the high-dimensional dexterous manipulation environments from Rajeswaran et al. (2018), which Nair et al. (2020) propose to use to study online adaptation with AWAC.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “[15] show that DAPG can eventually solve these tasks with more reward information, but this highlights the weakness of on-policy methods in sparse reward scenarios.”
Solving Rubik's Cube with a Robot Hand (2019, background): “There has also been significant progress in learning complex in-hand dexterous manipulation [84, 7], tool use [86] and even solving a smaller model of a Rubik’s Cube [62] using deep reinforcement learning, but those approaches were likewise only evaluated in simulation.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “, Atari [29]), this is significantly more challenging in real-world physical tasks, as it requires human intervention in the form of manual resets or the availability of a hard coded reset [28, 47, 37, 52, 21].”
Adaptive Online Planning for Continual Lifelong Learning (2019, background): “Similarly to past work [28, 33], we found that imitation learning can cap the asymptotic performance of the learned policy.”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Policy optimization In order to optimize the policy in the pessimistic MDP, we use the normalized natural policy gradient algorithm [32, 33, 76, 28].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “The original work using this dataset [Rajeswaran et al., 2018] was able to succesfully learn policies by combining imitation learning on the demonstrations with online RL, but found the demonstrations themselves insuﬃcient for reasonable success rates.”
A Framework for Data-Driven Robotics (2019, background): “Demonstrations: Demonstrations in RL have gained popularity in recent years [22], [23] as they help to address the exploration problem.”
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, background): “systems can be expensive and time-consuming due to their prohibitively high sample complexity [13], [14], [15].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “…followed by ﬁnetuning ( BC ), (2) ﬂat goal-conditioned behavior cloning trained with data relabeling followed by ﬁnetuning ( GCBC ) [22], and variants of these algorithms that augment the BC and GCBC ﬁne-tuning with losses as described in Rajeswaran et al. [28] - (3) DAPG-BC and (4) DAPG-GCBC .”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…et al., 2013, Chemali and Lazaric, 2015, Piot et al., 2014], and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function [Rajeswaran et al., 2017, Nair et al., 2018, Večerík et al., 2017, Hester et al., 2018, Gao et al., 2018].”
Implicit Behavioral Cloning (2021, background): “One route is by collecting on-policy data of the learned policy, and potentially either labeling with rewards to perform on-policy reinforcement learning (RL) [54, 55, 56] or labeling actions by an expert [2].”
Offline RL Without Off-Policy Evaluation (2021, methodology): “, 2016] and Adroit [Rajeswaran et al., 2017] tasks in the the D4RL benchmark suite [Fu et al.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…BC has been used as a regularization for policy optimization with DDPG [Lillicrap et al., 2015, Nair et al., 2018, Goecks et al., 2020] and the natural policy gradient [Kakade, 2001, Rajeswaran et al., 2017], but with additional sophistication through modified replay buffers and pre-training.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “The Adroit domain is a realistic simulation based on the Shadow Hand robot, ﬁrst proposed by Rajeswaran et al. (2017).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “We also consider the Adroit manipulation tasks (Rajeswaran et al., 2017), a collection of 4 tasks with a 24-DOF simulated hand.”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “While the combination of demonstrations and reinforcement has been studied extensively in single task problems [20, 38, 29, 22], this combination is particularly important in meta-learning contexts where few-shot learning of new tasks is simply not possible without demonstrations.”