Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

By Jakob N. Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H. S. Torr, Pushmeet Kohli, Shimon Whiteson. 2017

681 citations recorded by the source corpus.

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micro-management confirm that these methods enable the successful combination of experience replay with multi-agent RL.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “…of the policy parameters [36, 37], by enforcing the new policy to stay Importance Sampling for Reinforcement Learning In RL, IS is generally used for sample ef-ﬁciency purposes: in off-policy policy gradient methods, IS allows to re-use previously sampled sequences more than once [48, 44, 12].”
Counterfactual Multi-Agent Policy Gradients (2017, background): “The closest to our problem setting is that of Foerster et al. (2017), who also use a multi-agent representation and decentralised policies.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Foerster et al. (2017) addresses the issue of non-stationarity when using an experience replay with independent learners to some extent.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Experience replay has also been widely used in reinforcement learning (Mnih et al., 2015, 2013; Van Hasselt, Guez, & Silver, 2016; Lillicrap et al., 2016; Lin, 1992; Adam, Busoniu, & Babuska, 2011; Foerster et al., 2017; Kapturowski, Ostrovski, Dabney, Quan, & Munos, 2019; Atkinson et al., 2021).”
Online Class-Incremental Continual Learning with Adversarial Shapley Value (2020, methodology): “Experience Replay (ER) The research of ER and the important role of replay buffers has been well-established in the reinforcement learning area (Rolnick et al. 2019; Foerster et al. 2017).”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, methodology): “Previous work has attempted to address this by inputting other agent’s policy parameters to the Q function [35], explicitly adding the iteration index to the replay buffer, or using importance sampling [9], but these approaches had mixed empirical success.”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “The COMA model (Foerster et al. 2017a) was then proposed to tackle the challenge of multiagent credit assignment.”
The StarCraft Multi-Agent Challenge (2019, methodology): “…deep RL algorithms to single-agent and multi-agent versions of the micromanagement tasks has thus been steadily appearing (Usunier et al., 2016; Foerster et al., 2017, 2018a; Rashid et al., 2018; Nardelli et al., 2018; Hu et al., 2018; Shao et al., 2018; Foerster et al., 2018b) with the…”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “Omidshaﬁei et al. (2017) and Foerster et al. (2017) address learning stabilisation to some extent, but still learn decentralised value functions and do not allow for the inclusion of extra state information during training.”
Learning with Opponent-Learning Awareness (2017, background): “However, most of this recent work considers fully cooperative settings [13, 14, 34] and emergent”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “We demonstrate the effectiveness of our learning framework on the StarCraft II 1 unit micromanage-ment benchmark used in Foerster et al. (2017; 2018); Rashid et al. (2018); Samvelyan et al. (2019b).”
Deep Coordination Graphs (2019, methodology): “To address this, Foerster et al. (2017) show how to stabilize IQL when using experience replay buffers.”
The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning (2019, background): “[5] propose a novel approach to stabilize the experience replay buffer in DQNs by conditioning on when the samples were collected, thereby easing the non-stationarity of independent learning.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, methodology): “…our attention mechanism, we can implement a more general and ﬂexible form of a multi-agent baseline that, unlike the advantage function proposed in Foerster et al. (2017a), doesn’t assume the same action space for each agent, doesn’t require a global reward, and attends dynamically to other…”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “This problem is amplified in MA-DRL, where using ERMs often results in deprecated transitions being sampled during training [4, 18, 19].”
Lenient Multi-Agent Deep Reinforcement Learning (2017, background): “However, care is required when using ERMs for multi-agent deep reinforcement learning (MA-DRL), as stored transitions can become outdated when agents update their policies in parallel [9].”