FeUdal Networks for Hierarchical Reinforcement Learning

By Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Manfred Otto Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu. 2017

1,079 citations recorded by the source corpus.

We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels -- allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits -- in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “The same arguments that support hierarchical RL (Dayan & Hinton, 1993; Sutton et al., 1999; Vezhnevets et al., 2017) also encourage having a hierarchical decomposition of the reward model.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Among successful applications of HRL we highlight h-DQN (Kulkarni et al. 2016a) and Feu-dal Networks (Vezhnevets et al. 2017).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Machado et al. (2017) and Vezhnevets et al. (2017) proposed the use of options to generate decisive agents, avoiding the dithering commonly observed in random walks.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “To solve this problem, we consider a model-free, on-policy learning baseline, the advantage actor-critic (A2C) [93, 74, 66] and its hierarchical extension called Feudal Network (FN) [21, 90] (Figure 4).”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “They can also be readily extended to be compatible with hierarchical RL (Roderick et al., 2017; Vezhnevets et al., 2017).”
Relational Deep Reinforcement Learning (2018, background): “, [32]), and hierarchical RL and planning [33, 34] to allow structured representations and reasoning to translate more fully into structured behaviors.”
Dilated Recurrent Neural Networks (2017, background): “It is worth mentioning that, the recent proposed Dilated LSTM [23] can be viewed as a special case of our model, which only contains one dilated recurrent layer with ﬁxed dilation.”
Population Based Training of Neural Networks (2017, methodology): “…of deep reinforcement learning, showing how PBT can be used to optimise UNREAL (Jaderberg et al., 2016) on DeepMind Lab levels (Beattie et al., 2016), Feudal Networks (Vezhnevets et al., 2017) on Atari games (Bellemare et al., 2013), and simple A3C agents for StarCraft II (Vinyals et al., 2017).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Recent work has investigated deep hierarchical agents [1, 48, 16, 31], building on methods that explicitly design a hierarchy over actions [32, 12], or utilize options [43, 35].”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “Recent works in this research area include meta-learning [10, 30, 1], hierarchical learning [42, 41] and continual learning approaches [19, 36].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Solutions like the options framework [6, 10], HAM [11], max-Q [12], and feudal networks [13, 14] present promising algorithmic frameworks for HRL.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “, 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhnevets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills representing the lower-level policy.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “, 2018), hierarchy (Kulkarni et al., 2016; Vezhnevets et al., 2017) or meta-learning (Al-Shedivat et al.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Ahilan & Dayan (2019) propose a hierarchy in FeUdal (Dayan & Hinton, 1993; Vezhnevets et al., 2017) style for cooperative multi-agent tasks, where managers and workers are predeﬁned and workers are expected to achieve the goal generated by managers.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “In hierarchical MARL, different subtasks are chosen concurrently by all agents, whereas only a single subtask is chosen for each segment in single-agent hierarchical RL [4, 41].”
Multi-Agent Common Knowledge Reinforcement Learning (2018, methodology): “Despite using a hierarchical policy structure, MACKRL is not directly related to the family of hierarchical reinforcement learning algorithms (Vezhnevets et al., 2017), as it does not involve temporal abstraction.”
Hindsight Experience Replay (2017, background): “Learning simultaneously to perform multiple tasks has been also investigated for a long time in the context of Hierarchical Reinforcement Learning, e.g. Bakker and Schmidhuber (2004); Vezhnevets et al. (2017).”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “A parallel line of work runs through the hierarchical reinforcement learning (HRL) literature [9], with recent work [8, 27, 50, 58, 84] attempting to learn both lower-level policies as well as a higher-level policy that calls them.”
Long Text Generation via Adversarial Training with Leaked Information (2017, methodology): “Similar to (Vezhnevets et al. 2017), the gradient of the MANAGER module is defined as ∇adv θmgt = −QF (st, gt)∇θmdcos ( ft+c − ft, gt(θm) ) , (7)”