Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

By Ivaylo Popov, Nicolas Manfred Otto Heess, Timothy P. Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerík, Thomas Lampe, Yuval Tassa, Tom Erez, Martin A. Riedmiller. 2017

285 citations recorded by the source corpus.

Deep learning and reinforcement learning methods have recently been used to solve a variety of problems in continuous control domains. An obvious application of these techniques is dexterous manipulation tasks in robotics which are difficult to solve using traditional control theory or hand-engineered approaches. One example of such a task is to grasp an object and precisely stack it on another. Solving this difficult and practically relevant problem in the real world is an important long-term goal for the field of robotics. Here we take a step towards this goal by examining the problem in simulation and providing models and techniques aimed at solving it. We introduce two extensions to the Deep Deterministic Policy Gradient algorithm (DDPG), a model-free Q-learning based method, which make it significantly more data-efficient and scalable. Our results show that by making extensive use of off-policy data and replay, it is possible to find control policies that robustly grasp objects and stack them. Further, our results hint that it may soon be feasible to train successful stacking policies by collecting interactions on real robots.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Emergence of Locomotion Behaviours in Rich Environments (2017, background): “…prior works on continuous action spaces (e.g. [10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.g. [14, 15, 16]).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “Prior work has observed that increasing the number of gradient steps for DDPG tends to improve sample efficiency (Gu et al., 2017; Popov et al., 2017).”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “Orthogonal to our approach, recent improvements to DDPG include distributed methods (Popov et al., 2017), along with multi-step returns and prioritized experience replay (Schaul et al.”
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, background): “systems can be expensive and time-consuming due to their prohibitively high sample complexity [13], [14], [15].”
Hyperparameter Selection for Imitation Learning (2021, background): “However, in many cases it is impossible or impractical to design a reward function which captures the desired outcomes (Popov et al., 2017).”
StarCraft II: A New Challenge for Reinforcement Learning (2017, methodology): “The use of supervised data such as replays or human demonstrations has been successful in robotics [2, 24], the game of Go [18, 31], and Atari [10].”
Hindsight Experience Replay (2017, background): “For example, Popov et al. (2017) use a cost function consisting of ﬁve relatively complicated terms which need to be carefully weighted in order to train a policy for stacking a brick on top of another one.”
DeepMind Control Suite (2018, background): “It is also worth noting that we have not made a concerted effort to maximise data efficiency, for example by making many mini-batch updates using the replay buffer per step in the environment, as in Popov et al. (2017).”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “[35] demonstrate how to solve a stacking problem efficiently using a distributed variant of DPG.”
Distributed Distributional Deterministic Policy Gradients (2018, background): “Most interestingly, there have also been recent attempts to distribute updates for the DDPG algorithm, (e.g. Popov et al., 2017) and more generally in this work we build on work of (Horgan et al., 2018) for implementing distributed actors.”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, background): “Beyond selection of examples from a pre-determined set, recent work in robotics [2, 7, 10, 36] study learning tasks with interactive visuo-motor setups such as robotic arms.”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “This sparse reward can create difficulties for learning-based approaches [6]; on the other hand, non-sparse reward functions for such tasks might lead to undesired behaviors [7].”
Emergent Coordination Through Competition (2019, background): “On the other hand, shaping rewards tend to induce sub-optimal policies (Ng et al., 1999; Popov et al., 2017); We show in Figure 5 however that this is mitigated by coupling training with hyper-parameter evolution which adaptively adjusts the importance of shaping rewards.”
TriFinger: An Open-Source Robot for Learning Dexterity (2020, background): “A large part of the robotic reinforcement learning (RL) community has hence focused on simulation experiments (e.g. Haarnoja et al., 2018; Fujimoto et al., 2018; Popov et al., 2017; Mnih et al., 2016; Heess et al., 2017; Duan et al., 2016; Henderson et al., 2018).”
Residual Reinforcement Learning for Robot Control (2018, background): “For example, to train an agent to perform block stacking, each step can be encoded into the reward [14].”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, background): “Block Stacking: Prior work on block stacking either heavily relied on human demonstrations [4], [50], or required significant reward engineering [3], [51], and/or carefully designed curriculum [3] of reaching, picking and placing blocks.”
Motion Planning Networks: Bridging the Gap Between Learning-Based and Classical Motion Planners (2019, methodology): “DRL has solved various challenging robotic tasks using both model-based [28]–[30] and model-free [31]–[33] approaches.”