Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

By Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, Shimon Whiteson. Journal of machine learning research, 2020

1,503 citations recorded by the source corpus.

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

Subfields: Artificial Intelligence, Reinforcement Learning

View Paper: https://arxiv.org/abs/2003.08839

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “Nevertheless, previous work on SMAC have reported higher performance when the centralised state was being exploited during learning[19, 1] A direct comparison (see Table 5.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “These settings are the same to QMIX (Rashid et al., 2020).”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, methodology): “Comparison between QMIX-NPS and QMIX demonstrates that parameter sharing can, as documented (Foerster et al., 2018; Rashid et al., 2018), speed up training.”
Growing Action Spaces (2019, background): “A less restrictive representation, such as that proposed by (Rashid et al., 2018), could help, but we leave this direction to future work.”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “QMIX [108] considers the same problem as VDN does, and proposed an algorithm which is basically an improvement over VDN [132].”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “these actor-critic methods can represent is not restricted, in practice they significantly underperform value-based methods like QMIX [30] on the challenging StarCraft Multi-Agent Challenge (SMAC) [34] benchmark [32, 31].”
Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning (2020, methodology): “Later QMIX is proposed to overcome the limitations of VDN (Rashid et al., 2018).”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, methodology): “Value decomposition (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Rashid et al., 2020; Wang et al., 2021) =1 to ease the difﬁculty of learning the joint Q-funciton.”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, background): “However, despite its promise, other recent studies have found that QTRAN performs empirically worse than QMIX in complex MARL environments (Mahajan et al., 2019; Samvelyan et al., 2019; Rashid et al., 2020a).”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, methodology): “In the heterogeneous 2s3z and 1c3s5z, both QMIX and QPD could solve it.”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, background): “Recent work (Lowe et al., 2017; Foerster et al., 2018; Sunehag et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019) in multi-agent reinforcement learning (MARL) mainly adopts centralized training decentralized execution (CTDE) paradigm to mitigate non-stationarity.”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, methodology): “…work on this question (Bescuca, 2019) is inconclusive due to the confounder that the policy gradient methods studied (COMA and Central-V (Foerster et al., 2018)) are on-policy, while the respective Q - learning method, QMIX (Rashid et al., 2018), is off-policy with experience replay (Lin, 1992a).”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “A concurrent work (de Witt et al., 2020) ﬁnds that a decomposed centralized critic in QMIX style can improve the performance of MADDPG for learning in continuous action spaces.”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, methodology): “, recent deep methods [7–9, 19, 25, 26, 28, 30]), but many of these approaches assume centralized training and de-”