Feudal Reinforcement Learning

By Peter S. Dayan, Geoffrey E. Hinton. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Recursively Summarizing Books with Human Feedback (2021, methodology): “While Dayan and Hinton (1993) and Christiano et al. (2018) only tried this on simple algorithmic tasks, Perez et al. (2020) and Rajani et al. (2019) use similar ideas for question-answering tasks using a single step of decomposition.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “The same arguments that support hierarchical RL (Dayan & Hinton, 1993; Sutton et al., 1999; Vezhnevets et al., 2017) also encourage having a hierarchical decomposition of the reward model.”
Concrete Problems in AI Safety (2016, background): “Hierarchical reinforcement learning [39] offers another approach to scalable oversight.”
Neural Approaches to Conversational AI (2018, neutral): “mized with subgoals discovered by SDN is competitive to expert-designed subgoals. Finally, another interesting attempt is made by Casanueva et al. (2018) based on Feudal Reinforcement Learning (FRL) (Dayan and Hinton, 1993). In contrast to the above methods that decompose a task into temporally sepa”
Deep learning in neural networks: An overview (2014, methodology): “While HRL frameworks such as Feudal RL (Dayan & Hinton, 1993) and options (Barto, Singh, & Chentanez, 2004; Singh et al., 2005; Sutton, Precup, & Singh, 1999) do not directly address the problem of automatic subgoal discovery, HQ-Learning (Wiering & Schmidhuber, 1998a) automatically decomposes…”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “To solve this problem, we consider a model-free, on-policy learning baseline, the advantage actor-critic (A2C) [93, 74, 66] and its hierarchical extension called Feudal Network (FN) [21, 90] (Figure 4).”
Graying the black box: Understanding DQNs (2016, background): “Our methodology is to extract the neural activations of the last DQN hidden layer, while it is playing the game, and then apply t-Distributed Stochastic Neighbor Embedding (t-SNE) (Van der Maaten & Hinton, 2008) for dimensionality reduction and visualization.”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “of reach. However, option discovery where non-linear state approximations are required is still an open problem. Other related work for hierarchical formulations include the model of Dayan and Hinton [6] which consisted of \managers" taking decisions at various levels of granularity, percolatin”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “The goal of acquiring primitives for the purpose of enabling efficient learning on downstream tasks is also a central goal of hierarchical reinforcement learning (HRL), which has been a long-standing area of research [11, 51, 32, 43, 12].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Solutions like the options framework [6, 10], HAM [11], max-Q [12], and feudal networks [13, 14] present promising algorithmic frameworks for HRL.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Ahilan & Dayan (2019) propose a hierarchy in FeUdal (Dayan & Hinton, 1993; Vezhnevets et al., 2017) style for cooperative multi-agent tasks, where managers and workers are predeﬁned and workers are expected to achieve the goal generated by managers.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “Building on the framework of options, temporally-extended actions, and hierarchical single-agent RL [8, 26, 33, 38], early work on hier-”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016; Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).”
Reinforcement Learning: A Survey (1996, background): “Feudal Q-learning (Dayan & Hinton, 1993; Watkins, 1989) involves a hierarchy of learning modules.”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, background): “The original feudal reinforcement learning formulation of Dayan & Hinton (1993) advocated completely concealing the reward from the environment from lower levels of hierarchy.”
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition (1999, methodology): “The MAXQ method described in this paper follows this approach, building upon previous work by Singh (1992a), Kaelbling (1993), Dayan and Hinton (1993), and Dean and Lin (1995).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “While HRL frameworks such as Feudal RL [48] and options [275, 16, 261] do not directly address the problem of automatic subgoal discovery, HQ-Learning [309] automatically decomposes problems in partially observable environments into sequences of simpler subtasks that can be solved by memoryless poli”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Our work is related to FeUdal Networks (FuN) [43], originally inspired from feudal RL [7].”