FACMAC: Factored Multi-Agent Centralised Policy Gradients

By Bei Peng, Tabish Rashid, Christian Schröder de Witt, Pierre-Alexandre Kamienny, Philip H. S. Torr, Wendelin Böhmer, Shimon Whiteson. Neural Information Processing Systems, 2020

370 citations recorded by the source corpus.

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Public Health, Environmental and Occupational Health

View Paper: https://www.semanticscholar.org/paper/7e61f1787cbf97bb59eb7f5f383e654d66947f0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

A review of cooperative multi-agent deep reinforcement learning (2019, background): “In contrast to the DOP [229], where the centralized critic is factorized as a weighted sum of local critics (see (34)), [155] proposes a new framework that utilizes a nonmonotonic factorization for the centralized critic to improve the results for the tasks that cannot be solved using monotonic fact”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, neutral): “1) Safe Multi-Agent MuJoCo: Safe MAMuJoCo [74] 11 is an extension of MAMuJoCo [248].”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “Safe Multi-Agent MuJoCo Safe MAMuJoCo [99] 8 is an extension of MAMuJoCo [210].”