Hierarchical Deep Multiagent Reinforcement Learning

By Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, Li Wang. 2018

24 citations recorded by the source corpus.

Despite deep reinforcement learning has recently achieved great successes, however in multiagent environments, a number of challenges still remain. Multiagent reinforcement learning (MARL) is commonly considered to suffer from the problem of non-stationary environments and exponentially increasing policy space. It would be even more challenging to learn effective policies in circumstances where the rewards are sparse and delayed over long trajectories. In this paper, we study Hierarchical Deep Multiagent Reinforcement Learning (hierarchical deep MARL) in cooperative multiagent problems with sparse and delayed rewards, where efficient multiagent learning methods are desperately needed. We decompose the original MARL problem into hierarchies and investigate how effective policies can be learned hierarchically in synchronous/asynchronous hierarchical MARL frameworks. Several hierarchical deep MARL architectures, i.e., Ind-hDQN, hCom and hQmix, are introduced for different learning paradigms. Moreover, to alleviate the issues of sparse experiences in high-level learning and non-stationarity in multiagent settings, we propose a new experience replay mechanism, named as Augmented Concurrent Experience Replay (ACER). We empirically demonstrate the effects and efficiency of our approaches in several classic Multiagent Trash Collection tasks, as well as in an extremely challenging team sports game, i.e., Fever Basketball Defense.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “buffers B H , B L for both levels of the hierarchy, for off-policy updates in similar style to DQN [24], and initialize a dataset D for At the k -th high-level step, which occurs once for every t seg reward R t : = Algorithm 1 Hierarchical MARL with unsupervised skill discovery for each agent n do C”
Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces (2019, methodology): “To address the problem we mentioned in previous section, we propose another novel algorithm Deep MAHHQN, as inspired by Hierarchical Learning [ Kulkarni et al. , 2016; Tang et al. , 2018 ] .”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, background): “To the best of our knowledge, this hierarchical approach grounded in opponents’ state is unique; related works either do not assume a hierarchy of policies [9, 21, 14, 27], function with a centralised manager as top-level [1], or do not ground their hierarchy in game-theoretic fashion [33].”
Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning (2019, methodology): “The MAXQ algorithm [6] has been used to train homogenous agents [19], allowing them to coordinate by communicating subtasks rather than primitive actions, an idea recently re-explored in the context of deep RL [30].”
When Does Communication Learning Need Hierarchical Multi-Agent Deep Reinforcement Learning (2019, background): “The above-mentioned problems have been tackled by others extensively, using experience replay (Tang et al. 2018) or curriculum learning (Gupta, Egorov, and Kochenderfer 2017).”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “In [208] a cooperative problem with sparse and delayed rewards is considered, in which each agent accesses a local observation, takes a local action, and submits the joint action into the environment to get the local rewards.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Further utilization of the experience replay mechanism to cope with non-stationarity can be found in Tang et al. (2018) and Zheng et al. (2018a).”