QD-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations

By Soummya Kar, José M. F. Moura, H. Vincent Poor. 2012

205 citations recorded by the source corpus.

The paper considers a class of multi-agent Markov decision processes (MDPs), in which the network agents respond differently (as manifested by the instantaneous one-stage random costs) to a global controlled state and the control actions of a remote controller. The paper investigates a distributed reinforcement learning setup with no prior information on the global state transition and local agent cost statistics. Specifically, with the agents' objective consisting of minimizing a network-averaged infinite horizon discounted cost, the paper proposes a distributed version of $Q$-learning, $\mathcal{QD}$-learning, in which the network agents collaborate by means of local processing and mutual information exchange over a sparse (possibly stochastic) communication network to achieve the network goal. Under the assumption that each agent is only aware of its local online cost data and the inter-agent communication network is \emph{weakly} connected, the proposed distributed scheme is almost surely (a.s.) shown to yield asymptotically the desired value function and the optimal stationary control policy at each network agent. The analytical techniques developed in the paper to address the mixed time-scale stochastic dynamics of the \emph{consensus + innovations} form, which arise as a result of the proposed interactive distributed scheme, are of independent interest.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Stochastic Primal-Dual Q-Learning (2018, neutral): “We expect that this fundamental framework will be useful to advance many subﬁelds of RL, such as the distributed RL (Kar et al., 2013; Macua et al., 2015; Zhang et al., 2018; Lee et al., 2018), Q-learning with function approximations (Sutton et al., 2009b,a), sparsity promoted RL (Mahadevan and…”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “In [60, 59] the authors proposed a decentralized multi-agent version of the tabular Q-learning algorithm called QDlearning.”
Toward Massive Machine Type Communications in Ultra-Dense Cellular IoT Networks: Current Issues and Machine Learning-Assisted Solutions (2018, background): “Some existing works have applied collaborative Q-learning in different settings [25, 155, 157, 158].”
Stochastic Primal-Dual Q-Learning Algorithm For Discounted MDPs (2019, background): “Besides the direct advantage of theoretical guarantees, such optimization frameworks are also very favorable and extensible when dealing with constraints, sparsity regularizations [18], and distributed scenarios [19]–[22].”
Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning (2017, background): “As far as we know, all other previous works on distributed RL only considered tabular or linear function approximations (e.g., (Kar, Moura, and Poor 2013; Valcarcel Macua et al. 2015; Tutunov, Bou-Ammar, and Jadbabaie 2016)), and do not apply immediately to expressive nonlinear approximations.”
Social learning and distributed hypothesis testing (2014, background): “These “protocol” models study how message passing in a network can be used to achieve a pre-specified computational task such as distributed learning [16], [17], general function evaluation [18],or stochastic approximations [19].”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, methodology): “Moreover, these algorithms assume either the rewards are common to all agents (Boutilier, 1996; Lauer and Riedmiller, 2000), or there exists a remote central controller to take actions for the agents (Kar et al., 2013).”
Distributed Policy Evaluation Under Multiple Behavior Strategies (2013, methodology): “For example, the work in [24] proposes a useful algorithm, named QD-learning , which is a distributed implementation of Q-learning using consensus-based stochastic approxima tion.”
Applications of reinforcement learning in energy systems (2021, methodology): “Recently, several MARL algorithms, such as the distributed TD learning [207], distributed Q-learning [208], and distributed actor-critic algorithm [209], have been proposed by leveraging the core idea of averaging consensus-based distributed optimization [210].”
Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTD (2018, background): “The framework in [3] addresses the multiagent optimal policy design problem.”