Multi-Agent Common Knowledge Reinforcement Learning

By Jakob N. Foerster, Christian Schröder de Witt, Gregory Farquhar, Philip H. S. Torr, Wendelin Böhmer, Shimon Whiteson. 2018

119 citations recorded by the source corpus.

Cooperative multi-agent reinforcement learning often requires decentralised policies, which severely limit the agents' ability to coordinate their behaviour. In this paper, we show that common knowledge between agents allows for complex decentralised coordination. Common knowledge arises naturally in a large number of decentralised cooperative multi-agent tasks, for example, when agents can reconstruct parts of each others' observations. Since agents an independently agree on their common knowledge, they can execute complex coordinated policies that condition on this knowledge in a fully decentralised fashion. We propose multi-agent common knowledge reinforcement learning (MACKRL), a novel stochastic actor-critic algorithm that learns a hierarchical policy tree. Higher levels in the hierarchy coordinate groups of agents by conditioning on their common knowledge, or delegate to lower levels with smaller subgroups but potentially richer common knowledge. The entire policy tree can be executed in a fully decentralised fashion. As the lowest policy tree level consists of independent policies for each agent, MACKRL reduces to independently learnt decentralised policies as a special case. We demonstrate that our method can exploit common knowledge for superior performance on complex decentralised coordination tasks, including a stochastic matrix game and challenging problems in StarCraft II unit micromanagement.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “COMA (Foerster et al., 2018) instead uses a single centralised critic to train decentralised actors in the fully-cooperative setting, estimating a counterfactual advantage function for each agent in order to address multi-agent credit assignment.”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “StarCraft II [1] is a real-time strategy (RTS) game that has recently been utilized as a benchmark by the reinforcement learning community [16, 5, 15, 4].”
Learning Multi-Agent Communication through Structured Attentive Reasoning (2020, background): “Since CommNet, significant progress has been made in learning effective multi-agent communication (protocols) through the following methods: (i) broadcasting a vector representation of each agent’s private observations to all agents [14, 17]; (ii) selective and targeted communication through the use”
Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning (2020, methodology): “r the decentralized partially observable Markov decision process (DecPOMDP) [26], which describes fully cooperative multi-agent tasks. Speciﬁcally, we build on the setting of Dec-POMDPs with entities [29], introducing the concepts of entity types and scenarios. Our instantiation of a Dec-POMDP with ”
AI-QMIX: Attention and Imagination for Dynamic Multi-Agent Reinforcement Learning (2020, methodology): “Speciﬁcally, we build on the setting of Dec-POMDPs with entities [29], introducing the concepts of entity types and scenarios.”
CESMA: Centralized Expert Supervises Multi-Agents (2019, background): “d in [47]. A few works that follow the framework of centralized training, but decentralized execution are: RLar (Reinforcement Learning as Rehearsal) [16], COMA (Counterfactual Multi-Agent), and also [36, 11] – where the idea of knowledge-reuse is examined. In [6], they examine decentralization of p”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, background): “Significant progress has been made in cooperative multi-agent reinforcement learning (MARL) under the paradigm of centralised training with decentralised execution (CTDE) [27, 18] in recent years, both in value-based [41, 31, 37, 32, 44, 30] and actor-critic [23, 9, 36, 13, 8] approaches.”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “[36] considers a problem in which each agent observes a local observation, selects an action which is not known to other agents, and receives a joint reward, known to all agents.”
Rethinking Formal Models of Partially Observable Multiagent Decision Making (2019, background): “Indeed, we have recently seen breakthroughs in a number of problems that were made computationally feasible by decomposition based on public knowledge — examples include poker [28, 4, 5], Hanabi [15, 26], general EFGs [41, 27, 10], Dec-POMDPs [40, 44], and one-sided POSGs [18].”
Off-Belief Learning (2021, methodology): “Deep MARL has been successfully applied in many Dec-POMDP settings (de Witt et al., 2019; Foerster et al., 2019).”
Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning (2020, background): “However, the fully centralized critic of COMA suf-a fers difﬁculty in evaluating global Q-values from the joint state-action space especially when there are more than a handful of agents and is hard to give an appropriate multiagent baseline (Schroeder de Witt et al., 2019).”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “…is a diverse portfolio of multi-agent algorithms for cooperative tasks with discrete action spaces (Foerster et al., 2018; Rashid et al., 2018; Schroeder de Witt et al., 2019), decentralised continuous control algorithms have been largely limited to deep deterministic policy gradient…”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, unknown): “In this section, we present the evaluation of TMC on solving the StarCraft Multi-Agent Challenge (SMAC), a popular benchmark recently used by the MARL community [23, 39, 9, 22, 8].”