IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

By Lasse Espeholt, Hubert Soyer, R´emi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning and 2 more. 2018

1,890 citations recorded by the source corpus.

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “We test the Compressive Transformer as a drop-in replacement to an LSTM in the IMPALA setup (Espeholt et al., 2018).”
Environmental drivers of systematicity and generalization in a situated agent (2019, methodology): “Training algorithm We train the agent using an importance-weighted actor-critic algorithm with a central learner and distributed actors (Espeholt et al., 2018).”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “We refer to [50] for all other hyperparameter and implementation details.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “When training reinforcement learning (RL) agents, various levers exist to increase or stunt their performance: the choice of algorithms—e.g. A3C (Mnih et al., 2016) vs. IMPALA (Espeholt et al., 2018)—the number of training steps, the choice of training environments, the model capacity, the planning ”
Imitating Interactive Intelligence (2020, methodology): “Overall the setup was similar to IMPALA (Importance Weighted Actor-Learner Architectures) Espeholt et al. (2018).”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, methodology): “…to optimize policies for the trafﬁc and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torch-beast (K¨uttler et al., 2019), a PyTorch implementation of IMPALA (Espeholt et al., 2018), to optimize the policies for the Atari environment.”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “IMPALA [9] 191.”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[77] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “, 2016), distributed methods (Mnih et al., 2016; Espeholt et al., 2018), and approximate bounds (He et al.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “V-trace importance sampling is a popular off-policy correction for actor-critic agents (Espeholt et al., 2018).”
Model-Based Reinforcement Learning for Atari (2019, methodology): “, 2017) and actor-critic algorithms (Mnih et al., 2016; Schulman et al., 2017; Babaeizadeh et al., 2016; Wu et al., 2017; Espeholt et al., 2018).”
Experience Replay for Continual Learning (2018, methodology): “We employ distributed training based on the Importance Weighted Actor-Learner Architecture presented in [3].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Further improvements can be obtained by introducing control variates and clipping importance weights to control variance (Wang et al., 2016; Espeholt et al., 2018).”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, positive): “These notable successes were driven largely by on-policy RL algorithms such as IMPALA [10] and PPO [28, 4] which were often coupled with distributed training systems to utilize massive amounts of parallelism and compute.”
A Survey on Multi-Task Learning (2017, background): “Some works [155], [156], [157], [158], [159], [160] in MRL”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “A2C has been improved by IMPALA through parallel actors and correction for policy-lags between actors and learners [24].”
Meta-Gradient Reinforcement Learning (2018, methodology): “The n -step or λ -return can be augmented with off-policy corrections [Precup et al., 2000, Sutton et al., 2014, Espeholt et al., 2018] if it is necessary to correct for the distribution used to generate the data.”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “The setup for these experiments is similar to (Gregor et al., 2019), where the agent is trained using the IMPALA algorithm (Espeholt et al., 2018), and the model is trained alongside the agent via ELBO optimization on the data collected by the agent.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, background): “Recent work has demonstrated the benefit of distributed learning on a wide array of problems including single-player video games[48] and robotics[5].”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “3) we design and perform experiments to test the effects of both overshooting and memory, demonstrating that generative models benefit from these components more than deterministic models; 4) we show that training agents that share their belief-state with the generative model have substantially incr”