Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus

By Yan Zhang, Michael M. Zavlanos. 2019

51 citations recorded by the source corpus.

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional consensus step to let all the agents asymptotically achieve agreement on the global optimal policy function. The convergence analysis of the proposed algorithm is provided and the effectiveness of the proposed algorithm is validated using a distributed resource allocation example. Compared to relevant distributed actor critic methods, here the agents do not share information about their local tasks, but instead they coordinate to estimate the global policy function.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chongjie Zhang
2. Tonghan Wang
3. Bei Peng
4. Shimon Whiteson
5. Davood Hajinezhad
6. Afshin Oroojlooyjadid
7. Heng Dong
8. Beining Han

In Computational Theory and Mathematics

1. Davood Hajinezhad
2. Afshin Oroojlooyjadid
3. Chongjie Zhang
4. Zhuoran Yang
5. Kaiqing Zhang
6. Tamer Basar
7. Tonghan Wang
8. Anuj Mahajan

In Computer Networks and Communications

1. Afshin Oroojlooyjadid
2. Davood Hajinezhad
3. Tonghan Wang
4. Zhuoran Yang
5. Kaiqing Zhang
6. Tamer Basar
7. Shimon Whiteson
8. Beining Han

Citations in the corpus

• RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “Another off-policy algorithm for the same setting is proposed concurrently by [193], where agents do not share their estimates of value function.”
• Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
• A review of cooperative multi-agent deep reinforcement learning (2019, background): “In contrast to the [166], where the actor step is performed locally and the consensus update is proposed for the critic, in [167] the critic is performed locally, and the agents asymptotically achieve consensus on the actor parameter.”