Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

By Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, Tamer Basar. 2018

685 citations recorded by the source corpus.

We consider the problem of \emph{fully decentralized} multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

Citations in the corpus

Stochastic Primal-Dual Q-Learning (2018, neutral): “…that this fundamental framework will be useful to advance many subﬁelds of RL, such as the distributed RL (Kar et al., 2013; Macua et al., 2015; Zhang et al., 2018; Lee et al., 2018), Q-learning with function approximations (Sutton et al., 2009b,a), sparsity promoted RL (Mahadevan and Liu,…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games (2019, background): “…(MARL) (see Busoniu et al. (2008) for an earlier review), which recently has gained increasing research attention (see e.g. Foerster et al. (2016); Zhang et al. (2018a)), given the remarkable success of reinforcement learning in many applications, especially accompanied with the powerful function…”
Multi-Agent Reinforcement Learning: A Review of Challenges and Applications (2021, background): “In [29], the same setting of decentralized reinforcement learning with networked agents is addressed using an actor–critic algorithm.”
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning (2019, methodology): “nt, where m(s) is the emphatic weighting of s∈S, with vector form mT = dT µ(I− Pθ,γ)−1, where Pθ,γ ∈R |S|×| has entries given by Pθ,γ(s,s′) = γ X a∈A πθ(s,a)P(s′|s,a). Building on the work in [5] and [21], which themselves are built on [16], for the multi-agent case we obtain an expression for the o”
Multi-Agent Actor-Critic with Hierarchical Graph Attention Network (2019, positive): “Depending on the information available in the execution phase, CTDE approach can be further categorized into “learning-for-consensus approach”, where each agent determines decentralized action solely based on its local observation (Lowe et al. 2017; Foerster et al. 2018; Zhang et al. 2018; Iqbal and”
Applications of reinforcement learning in energy systems (2021, methodology): “Recently, several MARL algorithms, such as the distributed TD learning [207], distributed Q-learning [208], and distributed actor-critic algorithm [209], have been proposed by leveraging the core idea of averaging consensus-based distributed optimization [210].”
Graph Convolutional Reinforcement Learning (2018, background): “Sharing parameters of value function among agents is considered in (Zhang et al., 2018) and convergence guarantee is provided for linear function approximation.”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Tampuu et al. (2017) extended Deep Q-Network (DQN) (Mnih et al., 2013) into a multi-agent DQN for Pong game; Foerster et al. (2016); Sukhbaatar et al. (2016) explored communication protocol among agents; Zhang et al. (2018) further studied fully decentralized multi-agent system.”
Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTD (2018, methodology): “For example, it is not clear if there exists a theoretical guarantee that the proposed algorithm improves previous consensus algorithms [5], [6], [8].”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “First, we discuss the framework of extensive-form games in MARL, in addition to the conventional one of Markov games, or even simplified repeated games [11, 40, 20]; second, we summarize the progresses of a recently boosting subarea: decentralized MARL with networked agents, as an extrapolation of o”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “The main difference between this work and [166] comes from the fact that the critic step is conducted in an off-policy setting using emphatic temporal differences ET D ( λ ) policy evaluation method [134].”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “1 INTRODUCTION In recent years, cooperative multi-agent reinforcement learning is an active research area [4, 5, 14, 40] and has made exciting”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “…et al., 2019), optimizing both efﬁciency and fairness (Jiang & Lu, 2019), incorporating agent communication (Sukhbaatar et al., 2016; Das et al., 2019; Ding et al., 2020), considering relation between agents (Jiang et al., 2020) or their underlying network (Zhang et al., 2018; Qu et al., 2019).”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, background): “Zhang et al. [39] proposed decentralized actor-critic algorithms with function approximation, where each agent makes individual decisions based on both the information observed locally and the messages shared through a consensus step over the network.”