Scalable Multitask Policy Gradient Reinforcement Learning

By Salam El Bsat, Haitham Bou-Ammar, Matthew E. Taylor. 2017

21 citations recorded by the source corpus.

Policy search reinforcement learning (RL) allows agents to learn autonomously with limited feedback. However, such methods typically require extensive experience for successful behavior due to their tabula rasa nature. Multitask RL is an approach, which aims to reduce data requirements by allowing knowledge transfer between tasks. Although successful, current multitask learning methods suffer from scalability issues when considering large number of tasks. The main reasons behind this limitation is the reliance on centralized solutions. This paper proposes to a novel distributed multitask RL framework, improving the scalability across many different types of tasks. Our framework maps multitask RL to an instance of general consensus and develops an efficient decentralized solver. We justify the correctness of the algorithm both theoretically and empirically: we first proof an improvement of convergence speed to an order of O(1/k) with k being the number of iterations, and then show our algorithm surpassing others on multiple dynamical system benchmarks.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Distributed Multitask Reinforcement Learning with Quadratic Convergence (2018, methodology): “Recently, however, the authors in [11] proposed a distributed solver for MTRL with linear convergence guarantees based on the Alternating Direction Method of Multipliers (ADMM).”
Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning (2017, methodology): “3For simplicity, we assume that each agent is allocated with one task, similar to [7].”
Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “Multi-task reinforcement learning (MTRL) is a promising approach to train effective real-world agents (Tanaka & Yamamura, 2003; Rusu et al., 2016; Borsa et al., 2016; Rajeswaran et al., 2016; El Bsat et al., 2017; Andreas et al., 2017; Igl et al., 2020; D’Eramo et al., 2020; Yu et al., 2020a).”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, methodology): “Similar to prior works [9, 21, 32], we use distributed optimization to tackle the collective lifelong learning problem.”