Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning

By Sanjeevan Ahilan, Peter S. Dayan. 2019

92 citations recorded by the source corpus.

We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, we introduce Feudal Multi-agent Hierarchies (FMH). In this framework, a 'manager' agent, which is tasked with maximising the environmentally-determined reward function, learns to communicate subgoals to multiple, simultaneously-operating, 'worker' agents. Workers, which are rewarded for achieving managerial subgoals, take concurrent actions in the world. We outline the structure of FMH and demonstrate its potential for decentralised learning and control. We find that, given an adequate set of subgoals from which to choose, FMH performs, and particularly scales, substantially better than cooperative approaches that use a shared reward function.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Ahilan & Dayan (2019) propose a hierarchy in FeUdal (Dayan & Hinton, 1993; Vezhnevets et al., 2017) style for cooperative multi-agent tasks, where managers and workers are predeﬁned and workers are expected to achieve the goal generated by managers.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “buffers B H , B L for both levels of the hierarchy, for off-policy updates in similar style to DQN [24], and initialize a dataset D for At the k -th high-level step, which occurs once for every t seg reward R t : = Algorithm 1 Hierarchical MARL with unsupervised skill discovery for each agent n do C”
Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning (2019, background): “Deriving a successful communication protocol can already be seen as coordination in the communicative action space and can enable, to some extent, successful coordination in the physical action space [1].”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, background): “To the best of our knowledge, this hierarchical approach grounded in opponents’ state is unique; related works either do not assume a hierarchy of policies [9, 21, 14, 27], function with a centralised manager as top-level [1], or do not ground their hierarchy in game-theoretic fashion [33].”
When Does Communication Learning Need Hierarchical Multi-Agent Deep Reinforcement Learning (2019, background): “Many existing frameworks propose a separation of concerns for learning communication related tasks, e.g. Federated Control with Reinforcement Learning (Kumar et al. 2017), Feudal Multi-agent Hierarchies (Ahilan and Dayan 2019), and Master-Slave Multi-Agent Reinforcement Learning (Kong et al. 2017).”
Multi-agent deep reinforcement learning: a survey (2021, background): “Related to the idea of feudal networks (Dayan and Hinton 1993), Ahilan and Dayan (2019) applied a two-level abstraction of agents to a cooperative multi-agent setting where, in contrast to other methods, the hierarchy relied on rewards instead of state goals.”
RUMAD: Reinforcement-Unifying Multi-Agent Debate (2026, unknown): “HierarchicalRLframe-works like FMH [2] use multi-faceted rewards, where the objective is semantic task decomposition via manager-worker subgoaling.”