Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
By Sergio Valcarcel Macua, Aleksi Tukiainen, Daniel García-Ocaña Hernández, David Baldazo, Enrique Munoz de Cote, Santiago Zazo. 2017
31 citations recorded by the source corpus.
We propose a multiagent distributed actor-critic algorithm for multitask reinforcement learning (MRL), named \textit{Diff-DAC}. The agents are connected, forming a (possibly sparse) network. Each agent is assigned a task and has access to data from this local task only. During the learning process, the agents are able to communicate some parameters to their neighbors. Since the agents incorporate their neighbors' parameters into their own learning rules, the information is diffused across the network, and they can learn a common policy that generalizes well across all tasks. Diff-DAC is scalable since the computational complexity and communication overhead per agent grow with the number of neighbors, rather than with the total number of agents. Moreover, the algorithm is fully distributed in the sense that agents self-organize, with no need for coordinator node. Diff-DAC follows an actor-critic scheme where the value function and the policy are approximated with deep neural networks, being able to learn expressive policies from raw data. As a by-product of Diff-DAC's derivation from duality theory, we provide novel insights into the standard actor-critic framework, showing that it is actually an instance of the dual ascent method to approximate the solution of a linear program. Experiments illustrate the performance of the algorithm in the cart-pole, inverted pendulum, and swing-up cart-pole environments.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Chongjie Zhang
- Tonghan Wang
- Shimon Whiteson
- Bei Peng
- Davood Hajinezhad
- Afshin Oroojlooyjadid
- Heng Dong
- Zhuoran Yang
In Computational Theory and Mathematics
- Davood Hajinezhad
- Afshin Oroojlooyjadid
- Chongjie Zhang
- Zhuoran Yang
- Kaiqing Zhang
- Tamer Basar
- Tonghan Wang
- Anuj Mahajan
In Statistical and Nonlinear Physics
- Zhuoran Yang
- Shimon Whiteson
- Kaiqing Zhang
- Chongjie Zhang
- Tamer Basar
- Anuj Mahajan
- Davood Hajinezhad
- Afshin Oroojlooyjadid
Citations in the corpus
- RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
- Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
- A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “[84] proposes a new algorithm, called Diffusion-based Distributed Actor-Critic (Diff-DAC) for single and multi-task multi-agent RL.”
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, background): “One recent work (Macua et al., 2017) for multi-task RL, which is almost concurrent to ours, is also based on the model with networked agents.”