Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

By Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, Hongyao Tang. 2020

236 citations recorded by the source corpus.

In many real-world tasks, multiple agents must learn to coordinate with each other given their private observations and limited communication ability. Deep multiagent reinforcement learning (Deep-MARL) algorithms have shown superior performance in such challenging settings. One representative class of work is multiagent value decomposition, which decomposes the global shared multiagent Q-value $Q_{tot}$ into individual Q-values $Q^{i}$ to guide individuals' behaviors, i.e. VDN imposing an additive formation and QMIX adopting a monotonic assumption using an implicit mixing method. However, most of the previous efforts impose certain assumptions between $Q_{tot}$ and $Q^{i}$ and lack theoretical groundings. Besides, they do not explicitly consider the agent-level impact of individuals to the whole system when transforming individual $Q^{i}$s into $Q_{tot}$. In this paper, we theoretically derive a general formula of $Q_{tot}$ in terms of $Q^{i}$, based on which we can naturally implement a multi-head attention formation to approximate $Q_{tot}$, resulting in not only a refined representation of $Q_{tot}$ with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies. Extensive experiments demonstrate that our method outperforms state-of-the-art MARL methods on the widely adopted StarCraft benchmark across different scenarios, and attention analysis is further conducted with valuable insights.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “We compare our method with five state-of-the-art baselines: QTRAN [8], Qatten [9], QMIX [7], VDN [6], and independent Q-learning (IQL) [21] in the didactic and StarCraft II tasks.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “Inspired by [34] we then take the column-wise softmax of this weight matrix, which can be viewed as an approximation of multi-head attention.”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, background): “A large number of recent works [35, 46, 31, 41, 36] thus focus on developing new value-based MARL algorithms that address this representational limitation, in order to learn a richer class of action-value functions.”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, methodology): “To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018; Lowe et al., 2017; Du et al., 2019; Iqbal and Sha, 2019; Wang et al., 2020b) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020) have been proposed.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, methodology): “We drive the closed-form update rule of FQI-LVD, and then reveal the internal mechanism of credit assignment implicitly realized by linear value decomposition techniques, used by VDN and Qatten [19].”