The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning

By Jacopo Castellini, Frans Adriaan Oliehoek, Rahul Savani, Shimon Whiteson. 2019

45 citations recorded by the source corpus.

Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. In this work, we empirically investigate the representational power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results quantify how well various approaches can represent the requisite value functions, and help us identify issues that can impede good performance.

Subfields: Artificial Intelligence, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, neutral): “Castellini et al. (2019) investigate the representational capacity of coordination graphs in a deep RL setting using one-step matrix games.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “oordination is necessary, encoding those dependencies in a coordination graph. Both methods require the dependencies between agents to be pre-supplied, whereas we do not require such prior knowledge. Castellini et al. (2019) investigate the representational capacity of coordination graphs in a deep ”
Deep Coordination Graphs (2019, methodology): “…take inspiration from highly scalable methods (Yang et al., 2018; Chen et al., 2018) and improve over SCQL (Kok & Vlassis, 2006) and the approach of Castellini et al. (2019) by incorporating the following design principles: i. restricting the payoffs f ij ( a i , a j | τ it , τ ) to local…”
A survey and critique of multiagent deep reinforcement learning (2018, background): “coordination problems and how to learn those factorizations [244] (see Section 4.”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “…of the actors of MADDPG and MAAC has large variance due to the centralized critic (Wang et al., 2020), which is also The pathology of ﬁnding a sub-optimal solu-tion is also called relative overgeneralization (Wei & Luke, 2016; Castellini et al., 2019), which is discussed further in Appendix D.2.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “In value-based MARL, value decomposition (Guestrin et al., 2002b; Castellini et al., 2019) is widely used.”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, background): “On the other hand, GPL’s restrictive assumption that the learner’s joint action value function must factorize following a fully connected CG can degrade the learner’s returns for environments with certain reward functions (Castellini et al., 2019).”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, background): “Although these methods show promising results, they can face several challenges such as representational capacity [96] and inefficient exploration [97].”
Multi-agent deep reinforcement learning: a survey (2021, background): “Many approaches consider the decomposition of a joint value function into combinations of individual value functions (Castellini et al. 2019; Rashid et al. 2018; Son et al. 2019; Sunehag et al. 2018).”