Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU

By Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, Jan Kautz. 2016

297 citations recorded by the source corpus.

We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU's computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at this https URL .

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, neutral): “, 2017) and actor-critic algorithms (Mnih et al., 2016; Schulman et al., 2017; Babaeizadeh et al., 2016; Wu et al., 2017; Espeholt et al., 2018).”
Distributed Prioritized Experience Replay (2018, background): “There are a growing number of examples to justify this optimism: effective use of greater computational resources has been a critical factor in the success of such algorithms as Gorila (Nair et al., 2015), A3C (Mnih et al., 2016), GPU Advantage Actor Critic (Babaeizadeh et al., 2017), Distributed PP”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “arch groups) [212, 322].30 Within this context we propose two ways to address this problem. (1) Raising awareness: For DRL we found few works that study the computational demands of recent algorithms [335, 331]. For MDRL most published works do not provide information regarding computational resourc”
SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark (2018, background): “Prior approaches have relied on asynchronous SGD-style learning (e.g., A3C [27], Gorila [42], ADPG-R [28]), batched data collection for high GPU throughput (e.g., batched A2C [43], GA3C [44], BatchPPO [32]), and more recently, multiple CPU actors for experience generation and single GPU learner for ”
Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning (2018, methodology): “Its implementation was modified by [15] to efficiently use GPUs to maximize”
PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning (2018, background): “entropy term H(π(o)) to the policy loss, which has been shown to encourage exploration and discourage premature convergence [34] by penalizing a policy that always chooses the same actions.”
Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, methodology): “Comparing the training curves included in [6] for 3 common games tested in both works (Boxing, Breakout, SpaceInvaders) shows that our implementation is much faster and achieves as good or better scores 7 (see table 3).”
Deep Reinforcement Learning: An Overview (2017, methodology): “RL is a natural solution to some finance and economics problem s (Hull, 2014; Luenberger, 1997), like option pricing (Longstaff and Schwartz, 2001; Tsitsik lis and Van Roy, 2001; Li et al., 2009), and multi-period portfolio optimization (Brandt et al., 20 05), where value function based RL methods w”
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning (2018, methodology): “[7] extended A3C by proposing a CPU/GPU hybrid variation of A3C, GA3C, which moves the policy to the GPU.”
Teacher–Student Curriculum Learning (2017, background): “The agent code was separated into runners, which interact with the environment, and a trainer that performs batch training on GPU, similar to [40].”
Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning (2016, background): “Note that the contemporary work GA3C [Babaeizadeh et al. (2017)] also proposes a similar architecture.”
How to Close Sim-Real Gap? Transfer with Segmentation! (2020, methodology): “The training speed depends on the cost of generating training data, the sample efficiency of the learning algorithm Levine and Finn [2017], and the balance of the available computational resources Babaeizadeh et al. [2017]. Deep RL algorithms have been successfully used to play Go and Atari games at”
Accelerating Reinforcement Learning through GPU Atari Emulation (2019, methodology): “In the case of the CPU, scaling to multiple GPUs would be ineffective for on-policy algorithms, such GA3C [1, 2], or sub-optimal, in the case of distributed systems [6, 19].”
ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games (2017, methodology): “, BatchA3C [35] or GA3C [1]), where the model can perform batched forward prediction to better utilize GPUs.”
Core Challenges of Social Robot Navigation: A Survey (2021, methodology): “[61] relaxes prior assumptions with an actorcritic variant [8], simultaneously learning policies and agent motion models.”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, background): “GA3C (Babaeizadeh et al., 2017) recognized the potential of using a GPU in an asynchronous implementation for both action generation and learning.”