Multiagent Fully Decentralized Value Function Learning With Linear Convergence Rates

By Lucas Cassano, Kun Yuan, Ali H. Sayed. 2018

44 citations recorded by the source corpus.

This article develops a fully decentralized multiagent algorithm for policy evaluation. The proposed scheme can be applied to two distinct scenarios. In the first scenario, a collection of agents have distinct datasets gathered by following different behavior policies (none of which is required to explore the full state space) in different instances of the same environment and they all collaborate to evaluate a common target policy. The network approach allows for efficient exploration of the state space and allows all agents to converge to the optimal solution even in situations where neither agent can converge on its own without cooperation. The second scenario is that of multiagent games, in which the state is global and rewards are local. In this scenario, agents collaborate to estimate the value function of a target team policy. The proposed algorithm combines off-policy learning, eligibility traces, and linear function approximation. The proposed algorithm is of the variance-reduced kind and achieves linear convergence with $O(1)$ memory requirements. The linear convergence of the algorithm is established analytically and simulations are used to illustrate the effectiveness of the method.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chongjie Zhang
2. Tonghan Wang
3. Bei Peng
4. Shimon Whiteson
5. Heng Dong
6. Beining Han
7. Anuj Mahajan
8. Tarun Gupta

In Computational Theory and Mathematics

1. Chongjie Zhang
2. Tonghan Wang
3. Anuj Mahajan
4. Bei Peng
5. Beining Han
6. Shimon Whiteson
7. Heng Dong
8. Zongqing Lu

In Computer Networks and Communications

1. Tonghan Wang
2. Shimon Whiteson
3. Beining Han
4. Bei Peng
5. Heng Dong
6. Yihan Wang
7. Chongjie Zhang
8. Aadirupa Saha

Citations in the corpus

• RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
• Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “1 INTRODUCTION In recent years, cooperative multi-agent reinforcement learning is an active research area [4, 5, 14, 40] and has made exciting”
• Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”