MAVEN: Multi-Agent Variational Exploration

By Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, Shimon Whiteson. 2019

444 citations recorded by the source corpus.

Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “This results in signiﬁcant performance gains on some scenarios in SMAC (Samvelyan et al., 2019).”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, background): “However, current DRL systems still suffer from challenges of multi-agent coordination (Rashid et al., 2018; Mahajan et al., 2019; Yu et al., 2021), sparse rewards (Taiga et al.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Other methods, such as variational exploration (MAVEN [25]) and individuality emergence (EOI [26]), are slower to learn optimal strategies.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “More details can be found in appendix. well as popular exploration methods, i.e., count-based exploration (PG+CNT) (Tang et al., 2017) and MAVEN (Mahajan et al., 2019).”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “In this paper, we present empirical evidence for Independent PPO (IPPO), a multi-agent variant of proximal policy optimization [23], that shows IPPO matches or outperforms state-of-the-art MARL CTDE algorithms such as QMIX [19] or MAVEN [15] on multiple hard maps on SMAC [20].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “Although CTDE algorithms can solve many multi-agent problems (Mahajan et al., 2019; Das et al., 2019; Wang et al., 2020d), during training they must search in the joint action-observation space, which grows exponentially with the number of agents.”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “It is known that, these structures implement sufficient but not necessary conditions for the IGM constraint, which limit the representation expressiveness of joint action-value functions (Mahajan et al., 2019).”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “However, QMIX is unable to represent joint action value functions that are characterised as nonmonotonic [16], i.e., an agent’s ordering over its own actions depends on other agents’ actions.”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, methodology): “In order to facilitate decentralized control, we test our method on the SMAC benchmark (Samvelyan et al., 2019), which is the same as in (Foerster et al., 2017; 2018; Rashid et al., 2018; Mahajan et al., 2019).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “Our environment is multi-agent, and as such we face challenges of multiagent learning systems, characterised previously as nonstationarity, exploration, and interaction modelling (Bowling, 2000; Lowe et al., 2017; Mahajan et al., 2019).”
Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning (2019, background): “Mahajan et al. [25] also looks at sub-policies co-evolution to tackle the problem of joint exploration, however their selection mechanism occurs only on the ﬁrst timestep and requires duplicating random seeds across agents at test time.”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “The relaxation of these constraints makes QTRAN perform poorly in complex tasks (Mahajan et al., 2019).”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, unknown): “pacity [91] and inefficient exploration [92].”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, background): “Mahajan et al., (Mahajan et al., 2019) propose MAVEN to solve it, where value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy.”
QOPT: Optimistic Value Function Decentralization for Cooperative Multi-Agent Reinforcement Learning (2020, background): “[20] for evaluating state-of-the-art MARL methods.”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, background): “However, despite its promise, other recent studies have found that QTRAN performs empirically worse than QMIX in complex MARL environments (Mahajan et al., 2019; Samvelyan et al., 2019; Rashid et al., 2020a).”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, background): “This value-functional representation of agents’ actions enables studying interactions between them, as well to decompose the joint value function signal, thus helping alleviate the severity of the credit assignment problem [29, 35, 22].”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “Advances in valued-based MARL (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang et al., 2020e) contribute signiﬁcantly to the progress, achieving state-of-the-art performance on challenging tasks, such as StarCraft II micromanagement (Samvelyan et al., 2019).”