Asynchronous Methods for Deep Reinforcement Learning
By Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu. 2016
10,169 citations recorded by the source corpus.
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Pieter Abbeel
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
In Electrical and Electronic Engineering
- Sergey Levine
- Pieter Abbeel
- Shimon Whiteson
- Ofir Nachum
- Yoshua Bengio
- Chelsea Finn
- J. Urgen Schmidhuber
- Yuxi Li
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Ofir Nachum
- Yuxi Li
- Shixiang Shane Gu
- Abhishek Gupta
Citations in the corpus
- An Empirical Model of Large-Batch Training (2018, methodology): “For reinforcement learning, we use the following tasks via OpenAI Gym [BCP + 16]: • Dota 1v1 and 5v5 [BCD + 18] For Atari, we use A2C [MBM + 16] with a pure convolutional policy, adapted from OpenAI Baselines [DHK + 17].”
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “Similar weight-sharing techniques have also been explored in (Mnih et al., 2016).”
- Augmented Language Models: a Survey (2023, methodology): “Optimization is performed with the A3C algorithm (Mnih et al., 2016), a variant of the standard actor-critic method.”
- Deep Reinforcement Learning from Human Preferences (2017, methodology): “Recent success in scaling reinforcement learning (RL) to large problems has been driven in domains that have a well-specified reward function (Mnih et al., 2015, 2016; Silver et al., 2016).”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “The idea of using a slow-moving average target network to produce stable targets for the online network was inspired by deep RL [50, 51, 52, 53].”
- Representation Learning with Contrastive Predictive Coding (2018, methodology): “Here, we take the standard batched A2C [52] agent as base model and add CPC as an auxiliary loss.”
- Grounded Language Learning in a Simulated 3D World (2017, methodology): “This configuration offers a suitable trade-off between increased speed and loss of accuracy due to the asynchronous updates (Mnih et al., 2016).”
- Emergence of Locomotion Behaviours in Rich Environments (2017, methodology): “Second, like the widely used A3C algorithm [2] and related approaches [3] we distribute the computation over many parallel instances of agent and environment.”
- Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “r large action spaces. Policy gradient methods have been at the heart of significant advances in AI and robotics (Silver et al., 2014; Lillicrap et al., 2015; Silver et al., 2016; Levine et al., 2015; Mnih et al., 2016; Schulman et al., 2015a; Heess et al., 2015). Many of these methods are restricted”
- OpenAI Gym (2016, background): “…that combine deep learning with reinforcement learning have led to a great deal of excitement in the field, as it has become evident that general algorithms such as policy gradients and Q-learning can achieve good performance on difficult problems, without problem-specific engineering [2, 3, 4].”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “When training reinforcement learning (RL) agents, various levers exist to increase or stunt their performance: the choice of algorithms—e.g. A3C (Mnih et al., 2016) vs. IMPALA (Espeholt et al., 2018)—the number of training steps, the choice of training environments, the model capacity, the planning ”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “, 2015) instead of A3C (Mnih et al., 2016) to optimize the policy (see Appendix F).”
- Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “For the purpose of training the policy networks both within AGILE, and for our baseline trained from ground-truth reward rt instead of the modelled reward r̂t, we used the Asynchronous Advantage Actor-Critic (A3C; Mnih et al., 2016).”
- Constrained Policy Optimization (2017, background): “…network policies to achieve state-of-the-art performance on many high-dimensional control tasks, including Atari games (using pixels as inputs) (Mnih et al., 2015; 2016), robot locomotion and manipulation (Schulman et al., 2015; Levine et al., 2016; Lillicrap et al., 2016), and even Go at the…”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “The RL algorithm we use is a batched synchronous advantage actor-critic (A2C; Mnih et al. (2016)), or equivalently REINFORCE with baseline (Sutton and Barto, 2018); we found that V-MPO (Song et al., 2019) did not improve performance significantly and is computationally more expensive.”
- Red Teaming Language Models with Language Models (2022, methodology): “We train the red LM pr(x) with synchronous advantage actor-critic (A2C; Mnih et al., 2016).”
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Furthermore, we provide reliable implementations of popular on-policy RL algorithms that are tailored for LM fine-tuning such as PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015a), A2C (Mnih et al., 2016), and our own NLPO (§4).”
- Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Function approximation to learn a “critic” instead of using rewards directly has been embraced in the RL literature under the name of “actor-critic” methods (see Konda and Tsitsiklis (2000), Sutton et al. (2000), Kakade (2001), Schulman et al. (2015), Mnih et al. (2016), inter alia).”