A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
By Wesley A. Suttle, Zhuoran Yang, Kaiqing Zhang, Zhaoran Wang, Tamer Basar, Ji Liu. 2019
72 citations recorded by the source corpus.
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-agent version of emphatic temporal difference learning for off-policy policy evaluation, and proves convergence under linear function approximation. The paper then leverages this result, in conjunction with a novel multi-agent off-policy policy gradient theorem and recent work in both multi-agent on-policy and single-agent off-policy actor-critic methods, to develop and give convergence guarantees for a new multi-agent off-policy actor-critic algorithm.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tonghan Wang
- Davood Hajinezhad
- Afshin Oroojlooyjadid
- Chongjie Zhang
- Beining Han
- Heng Dong
- Yihan Wang
- Zhuoran Yang
In Computational Theory and Mathematics
- Davood Hajinezhad
- Afshin Oroojlooyjadid
- Zhuoran Yang
- Kaiqing Zhang
- Tamer Basar
- Chongjie Zhang
- Tonghan Wang
- Beining Han
In Computer Networks and Communications
- Afshin Oroojlooyjadid
- Davood Hajinezhad
- Tonghan Wang
- Zhuoran Yang
- Kaiqing Zhang
- Tamer Basar
- Beining Han
- Heng Dong
Citations in the corpus
- A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “In MARL literature, there exist a few algorithms utilizing the off-policy [165, 130, 83]; however, there is still room for research on off-policy in MARL in algorithm design, theory, and applications.”
- Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has achieved great progress in recent years (Hughes et al., 2018; Jaques et al., 2019; Vinyals et al., 2019; Zhang et al., 2019; Baker et al., 2020; Wang et al., 2020c).”
- Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “To address the nonconvergence issue in independent learning, agents are usually allowed to exchange/share local information with their neighbors over a communication network (Kar et al., 2013; Macua et al., 2015, 2017; Zhang et al., 2018,a,b; Lee et al., 2018; Wai et al., 2018; Doan et al., 2019; Su”