QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

By Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, Shimon Whiteson. 2018

2,072 citations recorded by the source corpus.

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Similar conclusions have been drawn in multi-agent domains: [24] report that multi-agent PG methods such as COMA are outperformed by MADDPG and QMix [25] by a clear margin in the particle-world environment [22] and the StarCraft multi-agent challenge [26].”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Since its introduction by Rashid et al. (2018), QMIX has become an important value-based algorithm in discrete action environments.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, positive): “In this experiment, we use QMIX (Rashid et al., 2018), MADDPG (Lowe et al., 2017), CDS (Li et al., 2021) and MAPPO (Yu et al., 2021) as the baselines.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Weighted QMIX [23] proposes weighted projection to decompose any joint action-value functions.”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, methodology): “, 2018), Q-Mix (Rashid et al., 2018), or Mean Field RL (Yang et al.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “…represents the action-value function with a deep neural network parameterized by θ. Mutli-agent Q-learning algorithms (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020) use a replay memory D to store the transition tuple (τ ,a, r, τ ′), where r is the reward…”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “In this paradigm of centralised training for decentralised execution, QMIX [25] is a popular Q - learning algorithm with state-of-the-art performance on the StarCraft Multi-Agent Challenge [26].”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “QMIX: QMIX [Rashid et al., 2018] extends VDN to address a broader class of environments.”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, background): “For example, Value-Decomposition-Networks (VDN) (Sunehag et al., 2017) represent the joint-Q-function as a sum of per-agent contributions and QMIX (Rashid et al., 2018) learns a non-linear but monotonic combination of these contributions.”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “Among value-based methods, value function factorization (Koller & Parr, 1999; Guestrin et al., 2002a; Sunehag et al., 2018; Rashid et al., 2018) methods have been proposed to efficiently handle a joint action-value function whose complexity grows exponentially with the number of agents.”
The StarCraft Multi-Agent Challenge (2019, methodology): “, 2018a) is an actor-critic algorithm with a special multi-agent critic baseline, and QMIX (Rashid et al., 2018) belongs to the Q-learning family.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “In this paradigm of centralized training with decentralized execution [5], a common approach [10, 27, 32, 36, 44] is to conduct centralized training at the level of primitive actions, which are the actions used in the transition function of the Markov game [20].”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “Our evaluation procedure is similar to [40, 43].”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “1 BASELINES AND ABLATIONS We compare our method with various baselines and ablations (i) QMIX (Rashid et al., 2018).”
Deep Coordination Graphs (2019, methodology): “QMIX uses a monotonic mixture of utilities instead of a sum (Rashid et al., 2018); (b) sum of pairwise payoffs”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “, win the battle in large-scale StartCraft II scenarios using existing methods (Samvelyan et al. 2019; Rashid et al. 2018).”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “We compare VBC and other banchmark algorithms, including VDN [18], QMIX [14] and SchedNet [8], for controlling allied units.”
Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, methodology): “Current deep multi-agent reinforcement learning algorithms mainly focus on learning in the multi-agent systems where agents are tightly coupled [Sukhbaatar et al. , 2016; Foerster et al. , 2018; Sunehag et al. , 2018; Rashid et al. , 2018].”