Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery

By Jiachen Yang, Igor Borovikov, Hongyuan Zha. 2019

101 citations recorded by the source corpus.

Human players in professional team sports achieve high level coordination by dynamically choosing complementary skills and executing primitive actions to perform these skills. As a step toward creating intelligent agents with this capability for fully cooperative multi-agent settings, we propose a two-level hierarchical multi-agent reinforcement learning (MARL) algorithm with unsupervised skill discovery. Agents learn useful and distinct skills at the low level via independent Q-learning, while they learn to select complementary latent skill variables at the high level via centralized multi-agent training with an extrinsic team reward. The set of low-level skills emerges from an intrinsic reward that solely promotes the decodability of latent skill variables from the trajectory of a low-level skill, without the need for hand-crafted rewards for each skill. For scalable decentralized execution, each agent independently chooses latent skill variables and primitive actions based on local observations. Our overall method enables the use of general cooperative MARL algorithms for training high level policies and single-agent RL for training low level skills. Experiments on a stochastic high dimensional team game show the emergence of useful skills and cooperative team play. The interpretability of the learned skills show the promise of the proposed method for achieving human-AI cooperation in team sports games.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “We compare with current state-of-the-art value-based MARL algorithms (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QPLEX (Wang et al., 2020b)), hierarchical MARL method (HSD (Yang et al., 2019)), and role-based MARL method (ROMA (Wang et al., 2020c)).”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, unknown): “Hierarchical frameworks (Yang et al., 2019; Kim et al., 2020; Xu et al., 2020) are also investigated to promote multi-agent cooperation/coordination.”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, methodology): “ be diverse but might not be useful, e.g., ignoring the pumping action. So it converges to a lower mean reward. We also train the primitive skills and high-level policy simultaneously, i.e., HC-simul [11], and ﬁnd it falls into local optimum. Before the diversity emerges in the skills, the high-leve”