QPLEX: Duplex Dueling Multi-Agent Q-Learning

By Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, Chongjie Zhang. 2020

633 citations recorded by the source corpus.

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or lead to poor performance. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “…to recent academic progress and literature in MARL which has largely focused developing off-policy learning frameworks such as MADDPG [22] and value-decomposed Q-learning [32, 27]; methods in these frameworks have yielded state-of-the-art results on a wide range of multi-agent benchmarks [36, 37].”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “CDS-All-Shared performs even worse than QPLEX, demonstrating that identity-aware diversity is difﬁcult to emerge without our specially designed network structure.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, methodology): “Recently, ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “FACMAC is also competitive with state-of-theart value-based methods (QMIX and QPLEX), with significantly better performance on MMM, bane_vs_bane, MMM2, and 27m_vs_30m.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “A drawback of an linearly decomposed critic is its limited representational capacity (Wang et al., 2020b), which may induce bias in value estimations.”
A survey of multi-agent deep reinforcement learning with communication (2022, positive): “Advantage functions can also replace the Q-function in the above equation to reduce variance [34].”