Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

By Shayegan Omidshafiei, Jason Pazis, Chris Amato, Jonathan P. How, John L. Vian. 2017

570 citations recorded by the source corpus.

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

A Survey on Multi-Task Learning (2017, background): “ a modular subpolicy is proposed to learn from policy sketches, which annotate tasks with sequences of named subtasks and provide information about high-level structural relationships among tasks. In [139], the problem of multi-task multi-agent reinforcement learning under the partial observability ”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “On the other hand, Omidshaﬁei et al. (2017) is one of the few efforts toward decentralized multi-agent learning which shows that optimistic learners can learn sample-efﬁcient MARL policies.”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “Most studied are cooperative settings, with strategies such as optimistic and hysteretic Q function updates [15, 21, 25], which assume that the actions of other agents are made to improve collective reward.”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, methodology): “Note that HDRQN has a large variance because it does not robustly solve the task—only a portion of seeds reached near-optimal policies.”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “This problem is amplified in MA-DRL, where using ERMs often results in deprecated transitions being sampled during training [4, 18, 19].”
Deep Reinforcement Learning for Swarm Systems (2018, background): “Omidshaﬁei et al. [2017] investigate hysteretic Q-learning [Matignon et al., 2007] and distillation [Rusu et al., 2015].”
Lenient Multi-Agent Deep Reinforcement Learning (2017, methodology): “Hysteretic Q-learning [17] is an algorithm designed for decentralised learning in deterministic multiagent environments, and which has recently been applied to MADRL as well [20].”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “However, with the notable exceptions of very recent work [22, 80], they have focused on (repeated) matrix games and/or the fully-observable case.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “Recently, the use of recurrent networks has been extended to MDRL to address the challenge of partially observability [158, 162, 164, 166, 180, 181, 170, 171, 174] for example, in FTW [179], depicted in Figure 5 and DRPIRQN [171] depicted in Figure 6.”
On Nonconvex Decentralized Gradient Descent (2016, background): “Some typical applications include: (i) the distributed compressed sensing problems [14], [30], [39], [45], [49]; (ii) distributed consensus [9], [29], [55], [58], [61], [69]; (iii) distributed and parallel machine learning [15], [21], [33], [43], [55].”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, neutral): “…extension, the multi-agent reinforcement learning (MARL), exemplified in Gupta et al. (2017); Lowe et al. (2017); Foerster et al. (2017a); Omidshafiei et al. (2017); Foerster et al. (2016); Sukhbaatar et al. (2016); Mordatch & Abbeel (2017); Havrylov & Titov (2017); Palmer et al. (2017);…”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, methodology): “Two learning rates 𝛼 and 𝛽 , named the increase rate and the decrease rate, are respectively used for updating overestimated and underestimated TD error 𝛿 : Hysteretic DQN (HDQN) [26] applies hysteresis to DQN, whose TD error is given by In practice, HDQN fixes the increase rate 𝛼 (e.g. 𝛼 = 0 .”
Federated Reinforcement Learning (2019, background): “To address this issue, (Omidshafiei et al., 2017) propose to explore Concurrent Experience Replay Trajectories (CERTs) structures, which store different agents’ histories, and align them together based on the episode indices and time steps.”
Generalized Hindsight for Reinforcement Learning (2020, background): “In the context of RL, transfer learning (Taylor & Stone, 2009) research has focused on learning transferable features across tasks (Parisotto et al., 2015; Barreto et al., 2017; Omidshafiei et al., 2017).”