Efficient Parallel Methods for Deep Reinforcement Learning

By Alfredo V. Clemente, Humberto Nicolás Castejón Martínez, Arjun Chandra. 2017

119 citations recorded by the source corpus.

We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at this https URL

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Ruslan Salakhutdinov
5. Shimon Whiteson
6. Yoshua Bengio
7. Trevor Darrell
8. Mohit Bansal

In Computational Theory and Mathematics

1. Shimon Whiteson
2. David Silver
3. Hado van Hasselt
4. Sergey Levine
5. Pieter Abbeel
6. Shangtong Zhang
7. Timothy P. Lillicrap
8. Nicolas Manfred Otto Heess

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Shimon Whiteson
4. David Silver
5. Hado van Hasselt
6. Timothy P. Lillicrap
7. Matteo Hessel
8. Chelsea Finn

In Efficiency & Compression

1. Karen Simonyan
2. Wendelin Böhmer
3. Pieter Abbeel
4. Silvio Savarese
5. Shimon Whiteson
6. Artyom Y. Sorokin
7. Shangtong Zhang
8. Koray Kavukcuoglu

In ML Systems & Hardware

1. Li Fei-Fei
2. Shimon Whiteson
3. Gabriel Barth-Maron
4. Wendelin Böhmer
5. Artyom Y. Sorokin
6. Shangtong Zhang
7. Iain Dunning
8. Mikhail S. Burtsev

Citations in the corpus

• Distributed Prioritized Experience Replay (2018, background): “GPU Asynchronous Actor-Critic (GA3C; Babaeizadeh et al., 2017) and Parallel Advantage Actor-Critic (PAAC; Clemente et al., 2017) adapt this approach to make efficient use of GPUs.”
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “One of the simplest of such methods is batched A2C (Clemente et al., 2017).”
• Generalized Off-Policy Actor-Critic (2019, methodology): “Hyperparameter Tuning: Our implementation is based on the A2C (Clemente et al., 2017) architecture.”
• Accelerated Methods for Deep Reinforcement Learning (2018, background): “In PAAC (Clemente et al., 2017), the authors explored the use of many simulators and increased batch sizes learning rates in (single-GPU) batched A2C– ideas central to our studies.”
• SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark (2018, background): “Prior approaches have relied on asynchronous SGD-style learning (e.g., A3C [27], Gorila [42], ADPG-R [28]), batched data collection for high GPU throughput (e.g., batched A2C [43], GA3C [44], BatchPPO [32]), and more recently, multiple CPU actors for experience generation and single GPU learner for ”
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory (2019, methodology): “To train the agent, we used the Parallel Advantage Actor Critic ( A2C ) algorithm (Mnih et al., 2016; Clemente et al., 2017).”