Distributed Distributional Deterministic Policy Gradients

By Gabriel Barth-Maron, Matthew W. Hoffman, David M. Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Manfred Otto Heess, Timothy P. Lillicrap. 2018

541 citations recorded by the source corpus.

This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of $N$-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.

Subfields: Artificial Intelligence, Biomedical Engineering, Cell Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Cell Biology

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Major improvements have originated from scaling deep RL to a distributed setting across many machines (Mnih et al., 2016; Schulman et al., 2017; Barth-Maron et al., 2018; Horgan et al., 2018; Espeholt et al., 2018; Anonymous, 2019a).”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “It signiﬁcantly outperforms the model-free algorithm A3C (Mnih et al., 2016) and in some cases D4PG (Barth-Maron et al., 2018) in terms of ﬁnal performance, while using 50 × less environment interaction and a similar amount of computation time.”
Solving Rubik's Cube with a Robot Hand (2019, background): “There has also been significant progress in learning complex in-hand dexterous manipulation [84, 7], tool use [86] and even solving a smaller model of a Rubik’s Cube [62] using deep reinforcement learning, but those approaches were likewise only evaluated in simulation.”
A Framework for Data-Driven Robotics (2019, methodology): “We use an algorithm similar to of D4PG [67] as our main training algorithm, but with a recurrent state.”
Regularized Behavior Value Estimation (2021, background): “…have been proposed, including variants with continuous actions (Lil-licrap et al., 2015; Haarnoja et al., 2018), distributional critics (Bellemare et al., 2017), prioritized replays (Schaul et al., 2015), and n-step returns (Kapturowski et al., 2019; Barth-Maron et al., 2018; Hessel et al., 2017).”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “Distributional perspective Since the introduction of distributional paradigm (see White (1988) and references therein) and its reincarnation for deep reinforcement learning (Bellemare et al., 2017) a great body of research emerged. Dabney et al. proposed a method to learn quantile values (or locatio”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “1 is represented as a discrete distribution, and the discrepancy measure d is the cross-entropy between the two distributions, as in Barth-Maron et al. (2018) and Wang et al. (2020).”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “D4PG Distributed Distributional Deep Deterministic Policy Gradient (Barth-Maron et al., 2018) is an online RL algorithm repurposed for offline RL. D4PG’s distributional critic estimates the distribution of discounted cumulative returns of the current policy, and its policy learns to take actions…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “The D4PG (Barth-Maron et al., 2018) and our novel DMPO agent are adaptations of DDPG and MPO, respectively, to use this distributional critic.”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, background): “• Q-Function Policy Gradient : DDPG [16], TD3 [17], SAC [18], ( coming soon : Distribu-tional DDPG [19]).”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, methodology): “Furthermore, distributional RL methods have shown, in single agent settings, robustness to hyperparameter variation and to have superior sample complexity and performance (Barth-Maron et al., 2018).”
Learning dexterous in-hand manipulation (2018, background): “There has also been significant progress in learning complex in-hand dexterous manipulation [49, 6] and even tool use [50] using deep reinforcement learning but those approaches were only evaluated in simulation as well.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, methodology): “Figure 5 compares the performance of DQN with ‘-greedy, bootstrap without prior (BS), bootstrap with prior networks (BSP) and the state-of-the-art continuous control algorithm D4PG, itself an application of ‘distributional RL’ [4].”
Playing hard exploration games by watching YouTube (2018, background): “Despite the recent advancements in deep reinforcement learning algorithms [7, 9, 17, 19] and architectures [22, 35], there are many “hard exploration” challenges, characterized by particularly sparse environment rewards, that continue to pose a difﬁcult challenge for existing RL agents.”
Data-Efficient Hierarchical Reinforcement Learning (2018, methodology): “In this way, we are able to use past experience for training the higher-level policy, taking advantage of progress made in recent years to provide stable, robust, and general off-policy RL methods [12, 28, 3].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “Reinforcement learning (RL) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-making tasks (Barth-Maron et al., 2018; Hessel et al., 2018; Nachum et al., 2019).”
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018, background): “The algorithm has been the basis for a large number of extensions [25, 26, 27, 28, 29, 30] which have further improved the performance of the agent.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “Baseline methods The highest reported performance on the continuous tasks is achieved by D4PG (Barth-Maron et al., 2018), an improved variant of DDPG (Lillicrap et al.”
Task-Relevant Adversarial Imitation Learning (2019, methodology): “Although many variants are introduced in the literature [33, 34, 4, 35, 36, 37], making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.”