Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

By Shixiang Shane Gu, Ethan Holly, Timothy P. Lillicrap, Sergey Levine. 2016

1,601 citations recorded by the source corpus.

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “…with an environment, and it has been used for applications such as playing games (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019; Team et al., 2021; Bakhtin et al., 2022) or controlling robots (Gu et al., 2017; Kalashnikov et al., 2018; Akkaya et al., 2019; Lee et al., 2020).”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “…prior works on continuous action spaces (e.g. [10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.g. [14, 15, 16]).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “Prior work has observed that increasing the number of gradient steps for DDPG tends to improve sample efficiency (Gu et al., 2017; Popov et al., 2017).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Broadly, these methods can be understood as trading off bias (Sutton et al., 1999; Silver et al., 2014; Lillicrap et al., 2016; Gu et al., 2016) and variance (Precup, 2000; Munos et al., 2016).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “This approximate gradient is used as a starting point in many widely used deep reinforcement learning algorithms (Silver et al., 2014; Lillicrap et al., 2015; Wang et al., 2016; Gu et al., 2017b).”
Solving Rubik's Cube with a Robot Hand (2019, neutral): “Alternative approaches include using many physical robots simultaneously, in order to be able to collect sufficient experience [36, 57, 49] or leveraging a model-based learning algorithms, which generally possess much more favorable sample complexity characteristics [121].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Gu et al. (2017) propose an asynchronous version of NAF.”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “uation using a simulator is not possible, such as in robotics [3] and dialogue systems [15, 30, 16]. Moreover, signiﬁcant efforts have been made to incorporate large-scale datasets into off-policy RL [17, 26, 9], but these works generally use large numbers of robots to collect online interaction dur”
Reward-estimation variance elimination in sequential decision processes (2018, background): “[3] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.”
Learning to Drive in a Day (2018, background): “In the recent years, reinforcement learning (RL) – a machine learning subfield focused on solving Markov Decision Problems (MDP) [3] where an agent learns to select actions in an environment in an attempt to maximise some reward function – has shown an ability to achieve super-human results at games”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “Recent results have shown success on games (Mnih et al. (2013)), locomotion (Schulman et al. (2015)), and a variety of robotic manipulation skills (Pinto & Gupta (2017); Schulman et al. (2016); Gu et al. (2017)).”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Recent years have seen reinforcement learning (RL) successfully applied to a number of robotics tasks such as in-hand manipulation [1], grasping [2] and door opening [3].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “In order to benchmark the capabilities of DRL with regard to the dexterous manipulation tasks outlined in Section III, we evaluate the NPG algorithm described briefly in Section V, and the DDPG algorithm [29], which has recently been used in a number of robotic manipulation scenarios [53], [13], [32”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “…a good policy, and is a particularly crucial ingredient in settings where interactions are costly or safety is of great importance, such as robotics (Gu et al., 2017; Kalashnikov et al., 2018a), health (Murphy et al., 2001), dialog agents (Jaques et al., 2019), and education (Mandel et al., 2014).”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “However, this sort of active online interaction is often impractical for real-world problems, where active data collection can be costly (Li et al., 2010), dangerous (Hauskrecht & Fraser, 2000; Kendall et al., 2019), or time consuming (Gu et al., 2017).”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “There has also been extensive work applying offline reinforcement learning to difficult real-world domains such as robots (Cabi et al., 2019; Gu et al., 2017; Kalashnikov et al., 2018) or dialog (Henderson et al., 2008; Jaques et al., 2019; Pietquin et al., 2011), but it is often difficult to do…”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, background): “Attempts for large scale data collection has also been pursued in rope manipulation [7], surgical robotics [22] and opening doors [23].”
Maximum a Posteriori Policy Optimisation (2018, background): “These algorithms exhibit much better data efﬁciency, approaching the regime where experiments with real robots are possible (Gu et al., 2016a; Andrychowicz et al., 2017).”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, methodology): “Deep RL has been applied to manipulation tasks [19], grasping [20], [21], opening a door [22], and locomotion [23], [24], [25].”