Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTD

By Donghwan Lee, Hyung-Jin Yoon, Naira Hovakimyan. 2018

49 citations recorded by the source corpus.

The goal of this paper is to study a distributed version of the gradient temporal-difference (GTD) learning algorithm for multi-agent Markov decision processes (MDPs). The temporal-difference (TD) learning is a reinforcement learning (RL) algorithm that learns an infinite horizon discounted cost function (or value function) for a given fixed policy without the model knowledge. In the distributed RL case each agent receives local reward through local processing. Information exchange over sparse communication network allows the agents to learn the global value function corresponding to a global reward, which is a sum of local rewards. In this paper, the problem is converted into a constrained convex optimization problem with a consensus constraint. We then propose a primal-dual distributed GTD algorithm and prove that it almost surely converges to a set of stationary points of the optimization problem.

Subfields: Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Stochastic Primal-Dual Q-Learning (2018, neutral): “…framework will be useful to advance many subﬁelds of RL, such as the distributed RL (Kar et al., 2013; Macua et al., 2015; Zhang et al., 2018; Lee et al., 2018), Q-learning with function approximations (Sutton et al., 2009b,a), sparsity promoted RL (Mahadevan and Liu, 2012), safe RL (Garcıa…”
Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus (2019, methodology): “The works in [12–16] employ distributed optimization methods to evaluate a given ﬁxed policy in multi-agents systems.”
Stochastic Primal-Dual Q-Learning Algorithm For Discounted MDPs (2019, background): “(b) Distributed RL: In distributed RL [22], each agent receives local reward through a local processing, while communicating over sparse and random networks to learn the global value function corresponding to the aggregate of local rewards.”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “Another variant of the distributed GTD algorithm was proposed in [73].”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “The use of MSPBE as an objective is standard in multi-agent policy evaluation (Macua et al., 2015; Lee et al., 2018; Wai et al., 2018; Doan et al., 2019), and the idea of saddle-point reformulation has been adopted in Macua et al.”