Value-Decomposition Networks For Cooperative Multi-Agent Learning

By Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls and 1 more. arXiv.org, 2017

1,350 citations recorded by the source corpus.

We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In the fully centralized and decentralized approaches, we find the problem of spurious rewards and a phenomenon we call the "lazy agent" problem, which arises due to partial observability. We address these problems by training individual agents with a novel value decomposition network architecture, which learns to decompose the team value function into agent-wise value functions. We perform an experimental evaluation across a range of partially-observable multi-agent domains and show that learning such value-decompositions leads to superior results, in particular when combined with weight sharing, role information and information channels.

Subfields: Reinforcement Learning

View Paper: https://arxiv.org/abs/1706.05296

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “An IQL agent (pre-trained with self-play) is trained sequentially with 5 partners (ordering denoted by arrows in Figure 2), and then evaluated with both its partners seen during continual training as well as some unseen agents (SAD+AUX+OP, VDN+AUX).”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “Value Decomposition Networks [25, VDN] decompose joint state-action value functions into sums of decentralised utility functions that can be used during greedy execution.”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “VDN: Value Decomposition Networks (VDN) [Sunehag et al., 2018] aim to learn a linear decomposition of the joint Q-value.”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, methodology): “We also train a joint-action Q-function that consists of the sum of per-agent Q-values to allow for off-policy learning in this multi-agent setting using Value Decomposition Networks (VDN) (Sunehag et al., 2017).”
The StarCraft Multi-Agent Challenge (2019, methodology): “Additionally, IQL, VDN, and QMIX all signiﬁcantly outperform COMA, demonstrating the sample efﬁciency of off-policy value-based methods over on-policy policy gradient methods.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “In between these two extremes, we can learn a centralised but factored Q tot , an approach taken by value decomposition networks (VDN) (Sunehag et al., 2017).”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “[46] propose Value Decomposition Networks (VDN), which learn the joint-action Q-values by factoring them as the sum of each agent’s Q-values.”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “We compare VBC and other banchmark algorithms, including VDN [18], QMIX [14] and SchedNet [8], for controlling allied units.”
Multi-Agent Common Knowledge Reinforcement Learning (2018, background): “Rashid et al. (2018) and Sunehag et al. (2017) propose learning a centralised value function that factors into per-agent components to allow for decentralisation.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “…& Abbeel, 2017; Lowe et al., 2017; Foerster et al., 2016), credit assignment in cooperative settings with global rewards (Rashid et al., 2018; Sunehag et al., 2017; Foerster et al., 2017a), and the failure to take opponent strategies into account when learning agent policies (He et al.,…”
Factorized Q-learning for large-scale multi-agent systems (2018, background): “Value-Decomposition Networks (VDN) [21] goes a little bit beyond IQL by summing over all the independent Q-functions for cooperative tasks, but the complex interactions in MARL are unlikely to be captured by simplistic linear summations.”
Deep Reinforcement Learning for Swarm Systems (2018, background): “Sunehag et al. [2017] tackle the “lazy agent” problem in cooperative MARL with a single team reward by training each agent with a learned additive decomposition of a value function based on the team reward.”
Lenient Multi-Agent Deep Reinforcement Learning (2017, methodology): “A recent successful approach has been to decompose a team value function into agentwise value functions through the use of a value decomposition network architecture [26].”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, methodology): “On average, TMC+VDN achieves 1.24× and 1.35× higher reward than the rest algorithms on PP and CN, respectively.”
Inequity aversion improves cooperation in intertemporal social dilemmas (2018, background): “Moreover naïvely optimizing group reward is also ineffective, due to the lazy agent problem [24].”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, background): “Significant progress has been made in cooperative multi-agent reinforcement learning (MARL) under the paradigm of centralised training with decentralised execution (CTDE) [26, 16] in recent years, both in value-based [38, 30, 35, 31, 41, 29] and actor-critic [21, 7, 11, 6] approaches.”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, background): “Recent work (Lowe et al., 2017; Foerster et al., 2018; Sunehag et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019) in multi-agent reinforcement learning (MARL) mainly adopts centralized training decentralized execution (CTDE) paradigm to mitigate non-stationarity.”
CESMA: Centralized Expert Supervises Multi-Agents (2019, background): “Namely, Also see (Sunehag et al., 2017; Rashid et al., 2018) where they provide more analysis in decomposing the Q -values in this way.”