Weighted QMIX: Expanding Monotonic Value Function Factorisation

By Tabish Rashid, Gregory Farquhar, Bei Peng, Shimon Whiteson. 2020

490 citations recorded by the source corpus.

QMIX is a popular $Q$-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action $Q$-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the $Q$-learning targets and then projects them into the space representable by QMIX. This projection returns a representable $Q$-value that minimises the unweighted squared error across all joint actions. We show in particular that this projection can fail to recover the optimal policy even with access to $Q^*$, which primarily stems from the equal weighting placed on each joint action. We rectify this by introducing a weighting into the projection, in order to place more importance on the better joint actions. We propose two weighting schemes and prove that they recover the correct maximal action for any joint action $Q$-values, and therefore for $Q^*$ as well. Based on our analysis and results in the tabular setting we introduce two scalable versions of our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW) QMIX and demonstrate improved performance on both predator-prey and challenging multi-agent StarCraft benchmark tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “• SMAC: QMix [27] and SOTA methods including QPlex [36], CWQMix [26], AIQMix [18] and RODE [37].”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Weighted QMIX [23] proposes weighted projection to decompose any joint action-value functions.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “These settings are the same to QMIX (Rashid et al., 2020).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “these actor-critic methods can represent is not restricted, in practice they significantly underperform value-based methods like QMIX [30] on the challenging StarCraft Multi-Agent Challenge (SMAC) [34] benchmark [32, 31].”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, methodology): “Value decomposition (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Rashid et al., 2020; Wang et al., 2021) =1 to ease the difﬁculty of learning the joint Q-funciton.”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, methodology): “Value decompositionmethods are widely used in value-basedMARL algorithms [16, 27].”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, methodology): “For Weighted QMIX (Rashid et al., 2020b), we additionally use a feed-forward network with 3 hidden layers of 128 dim and ReLU non-linearities and set α = 0 .”