Accelerated Methods for Deep Reinforcement Learning

By Adam Stooke, Pieter Abbeel. 2018

143 citations recorded by the source corpus.

Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.

Subfields: Artificial Intelligence, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

Continual Reinforcement Learning in 3D Non-stationary Environments (2019, methodology): “For all the experiments we use a simple batched-A2C with synchronous updates [47], but only within the same map (the actual environment with fixed static settings), so that when the map changes the model cannot access in any way previous environmental conditions.”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “Prior works [40, 41] have explored new reinforcement learning algorithms that allow compute parallelization to accelerate training.”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, methodology): “Previous experience in [23] found good scaling of asynchronous, multi-GPU A3C and PPO on Atari using a CPU parameter store, but this technique did not scale as well to larger networks with more training updates, such as in DQN.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Similarly, Mordatch and Abbeel [57] investigated the emergence of language with the difference that in their setting there were no explicit roles for the agents (i.e., sender, receiver).”
Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, methodology): “We build upon [10, 12] by pushing the parallelization further, optimizing the training algorithm, and applying the approach to a challenging real-world robotics task.”
Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, methodology): “e ones achieved by A3C (which was trained for 24 hours with a single CPU) [24, p. 7]. Detailed analysis of Deep Reinforcement Learning on a single machine with multiple GPUs was recently published in [31]. The authors reported using a batch size of up to 2048 and utilizing 8 GPUs for various Deep RL”
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning (2018, background): “Stooke and Abbeel [16] explored optimizing existing RL algorithms for fast performance on a multi-GPU system.”
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames (2019, background): “Stooke & Abbeel (2018) experiment with Atari and ﬁnd it not effective however.”
Accelerating Reinforcement Learning through GPU Atari Emulation (2019, methodology): “We first divide DRL algorithms into policy gradient and Q-value methods, as in [19].”