Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward

By Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls and 1 more. 2018

895 citations recorded by the source corpus.

We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In the fully centralized and decentralized approaches, we find the problem of spurious rewards and a phenomenon we call the "lazy agent'' problem, which arises due to partial observability. We address these problems by training individual agents with a novel value-decomposition network architecture, which learns to decompose the team value function into agent-wise value functions.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “• Hanabi : SAD [15] and VDN [32].”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Sunehag et al. (2017) propose value decomposition networks (VDN), which allow for centralised value function learning with decentralised execution.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “VDN [4], QMIX [5], and QTRAN [21] progressively expand the representation capabilities of the mixing network.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “We compare with current state-of-the-art value-based MARL algorithms (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QPLEX (Wang et al., 2020b)), hierarchical MARL method (HSD (Yang et al., 2019)), and role-based MARL method (ROMA (Wang et al., 2020c)).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “Training with Offline Data Collection To construct a diverse dataset, we train a behavior policy of QMIX (Rashid et al., 2018) or VDN (Sunehag et al., 2018) and collect its 20k or 50k experienced episodes throughout the training process.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “By contrast VDN [28] and QMIX [25] learn the joint-action Q -values, which avoids some of these issues.”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “Three representative examples: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) gradually increase the representation ability of the mixing network.”
Deep Coordination Graphs (2019, methodology): “Sunehag et al. (2018) introduces parameter sharing between the agents’ utility functions to dramatically improve the sample efﬁciency of VDN. Agents can have different action spaces A i but the choice of unavailable actions during maximization can be prevented by setting the utilities of…”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Figure5(d-f) depict the average win rate of VDN with and without our DyMA-CL.”
Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, background): “Current deep multi-agent reinforcement learning algorithms mainly focus on learning in the multi-agent systems where agents are tightly coupled [Sukhbaatar et al. , 2016; Foerster et al. , 2018; Sunehag et al. , 2018; Rashid et al. , 2018].”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, methodology): “Sunehag et al. (2018) designed a value-decomposition network (VDN) to learn an optimal linear value decomposition from the team reward signal based on the assumption that the joint action-value function for the system can be additively decomposed into value functions across agents.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “ it can better address the long-term credit assignment challenge for RL problems. Recently, the use of recurrent networks has been extended to MDRL to address the challenge of partially observability [158, 162, 164, 166, 180, 181, 170, 171, 174] for example, in FTW [179], depicted in Figure 5 and DR”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, methodology): “VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) have progressively enlarged the family of functions that can be represented by the mixing network.”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, methodology): “To coordinate this exchange of messages, we adopt the centralized training and distributed execution paradigm popularized in recent works, e.g. , Foerster et al. (2017a); Lowe et al. (2017); Sunehag et al. (2018); Rashid et al. (2018); Gupta et al. (2017).”
"Other-Play" for Zero-Shot Coordination (2020, background): “Furthermore, note that the OP policy in this setting is the only policy that would never be chosen by the types of algorithms that try to use self-play to optimize team performance, e.g. VDN (Sunehag et al., 2018) or SAD (Hu & Foerster, 2019).”