Counterfactual Multi-Agent Policy Gradients

By Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson. AAAI Conference on Artificial Intelligence, 2017

2,656 citations recorded by the source corpus.

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

View Paper: https://arxiv.org/abs/1705.08926

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Toward Causal Representation Learning (2021, background): “3) Counterfactuals: Counterfactual reasoning has been found to improve the data efﬁciency of RL algorithms [38], [164] and improve performance [51], and it has been applied to communicate about past experiences in the multiagent setting [69], [241].”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Several past CTDE methods [21, 11] adopt actor-critic structures and learn a centralized critic which takes global information as input.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “At the other extreme, we can learn a fully centralised action-value function Qtot and then use it to guide the optimisation of decentralised policies in an actor-critic framework, an approach taken by counterfactual multi-agent (COMA) policy gradients (Foerster et al., 2018), as well as work by Gupt”
Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, positive): “MARL has benefited from recent developments in deep reinforcement learning, with the frameworks moving away from tabular methods [2] to end-to-end schemes [7].”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “inforcement Learning [5, 38] in cooperative settings has been widely studied [11, 12, 15, 23, 28, 37], and applied to domains such as video games [31] and visual question answering [9].”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, methodology): “Concurrently to our work, [7] proposed a similar idea of using policy gradient methods with a centralized critic, and test their approach on a StarCraft micromanagement task.”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “The COMA model (Foerster et al. 2017a) was then proposed to tackle the challenge of multiagent credit assignment.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “COMA [20], MADDPG [9], PR2 [27], and DOP [10] study the problem of policy-based multi-agent reinforcement learning.”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, background): “While the use of FuPS is limited and our expectations are not high, since there is no way to differentiate between agents, FuPS+id is encountered very often in the literature (Rashid et al., 2018; Foerster et al., 2018; Gupta et al., 2017).”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “While easy to distribute and decentralisable by construction, IL suffers from a variety of theoretical limitations that may result in learning instabilities or suboptimal performance observed in practice [27, 9, IQL, IAC].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “COMA (Foerster et al., 2018) and MADDPG (Lowe et al., 2017) propose the paradigm of centralized critic with decentralized actors (CCDA).”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “For the actor’s policies we use the same (cid:15) -greedy ﬂoor technique as in [9].”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “…applies single-agent RL algorithms for each agent without consideration of the multi-agent structure [Tan, 1993]; centralised multi-agent policy gradient [Lowe et al., 2017, Foerster et al., 2018, Yu et al., 2021]; and value decomposition [Sunehag et al., 2018, Rashid et al., 2018] algorithms.”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, background): “The typical workout for this is a public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer to Moravčík et al. (2017); Nayyar et al. (2013); Foerster et al. (2018b) for more details.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “At optimization time, the value function has access to the full unmasked environment state, similar to [59, 33, 60].”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “More recent studies, such as COMA (Foerster et al., 2018), take advantage of CTDE (Oliehoek et al., 2008); actors are trained by a joint critic to estimate a counterfactual baseline designed to gauge each agent’s contribution to the shared task.”
The StarCraft Multi-Agent Challenge (2019, methodology): “We utilise the same -ﬂoor scheme as in (Foerster et al., 2018a) for the agents’ policies, linearlly annealing (cid:15) from 0.5 to 0.01 over 100k timesteps.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “At the other extreme, we can learn a fully centralised state-action value function Q tot and then use it to guide the optimisation of decentralised policies in an actor-critic framework, an approach taken by counterfactual multi-agent (COMA) policy gradients (Foerster et al., 2018), as well as work ”
Learning with Opponent-Learning Awareness (2017, background): “However, most of this recent work considers fully cooperative settings [13, 14, 34] and emergent”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “choose from a diverse set of cooperative MARL algorithms with decentralized execution [10, 32, 36, 44].”