Reinforcement Learning with Hierarchies of Machines

By Ronald E. Parr, Stuart J. Russell. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Many real-world tasks exhibit hierarchical structure, where higher level decisions can reuse lower level skills (Parr & Russell, 1998; Sutton et al., 1999; Dietterich, 2000).”
Neural Approaches to Conversational AI (2018, neutral): “(2010) proposed to use hierarchical reinforcement learning to optimize a composite task’s dialogue policy, with tabular versions of the MAXQ (Dietterich, 2000) and Hierarchical Abstract Machine (Parr and Russell, 1998) approaches.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “this, continual learning agents should reuse aspects of the solutions to previously solved subproblems through function composition (Griffiths, Callaway, Chang, Grant, Krueger, and Lieder, 2019) by abstracting relevant meaningful information in the form of abstract concepts (Parr and Russell, 1998) ”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “The goal of acquiring primitives for the purpose of enabling efficient learning on downstream tasks is also a central goal of hierarchical reinforcement learning (HRL), which has been a long-standing area of research [11, 51, 32, 43, 12].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Solutions like the options framework [6, 10], HAM [11], max-Q [12], and feudal networks [13, 14] present promising algorithmic frameworks for HRL.”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, neutral): “…learning (Sutton et al., 1999; Precup, 2000; Dayan & Hinton, 1993; Dietterich, 2000; Boutilier et al., 1997; Dayan, 1993; Kaelbling, 2014; Parr & Russell, 1998; Precup et al., 1997; 1998; Schmidhuber, 1991; Sutton, 1995; Wiering & Schmidhuber, 1997; Vezhnevets et al., 2016; Bacon et al.,…”
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition (1999, methodology): “In addition to the three domains discussed above, we have developed MAXQ graphs for Singh’s (1992b) “flag task”, the treasure hunter task described by Tadepalli and Dietterich (Tadepalli & Dietterich, 1997), and Dayan and Hinton’s (1993) Fuedal-Q learning task.”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “In all cases, our approach substantially outperforms previous approaches based on explicit decomposition of the Q function along subtasks (Parr & Russell, 1998; Vogel & Jurafsky, 2010), unsupervised option discovery (Bacon & Precup, 2015), and several standard policy gradient baselines.”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, neutral): “For these problems, RL needs hierarchy to succeed [Parr and Russell, 1998], which is not as necessary when using black box optimization.”
IQA: Visual Question Answering in Interactive Environments (2017, neutral): “Early works proposing HRL algorithms include [11, 54, 64].”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Typically these have the form of a meta controller and a set of lower-level controllers for subtasks (Sutton et al., 1999; Dietterich, 2000; Parr and Russell, 1997; Ghavamzadeh and Mahadevan, 2003; Konidaris et al., 2012; Konidaris and Barto, 2007).”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Hierarchical reinforcement learning (HRL), in which multiple layers of policies are trained to perform decision-making and control at successively higher levels of temporal and behavioral abstraction, has long held the promise to learn such difficult tasks [7, 29, 39, 4].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “To tackle these challenges, two main strategies have been pursued: The ﬁrst strategy is to design a hierarchy over the actions (Parr & Russell, 1998; Sutton et al., 1999; Dietterich, 2000).”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “2 Related Work Learning temporally extended actions and temporal abstraction in general are long standing problems in reinforcement learning [5, 6, 7, 8, 12, 20, 21, 22, 23, 25, 27, 28, 30].”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, background): “…has long held the promise of extending the successes of existing reinforcement learning (RL) methods (Gu et al., 2017; Schulman et al., 2015; Lillicrap et al., 2015) to more complex, difﬁcult, and temporally extended tasks (Parr & Russell, 1998; Sutton et al., 1999; Barto & Mahadevan, 2003).”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “Hierarchical reinforcement learning is a well studied area in reinforcement learning (Sutton et al., 1999; Dayan & Hinton, 1992; Schmidhuber, 2008; Parr & Russell, 1997; Dietterich, 1998).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Designing, discovering and learning meaningful and effective abstractions of MDPs has been studied extensively in hierarchical reinforcement learning (HRL) [15, 40, 57, 16, 4].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “One class of methods (Bacon et al., 2017; Frans et al., 2017; Kaelbling, 1993a; Kulkarni et al., 2016; Nachum et al., 2018; Parr and Russell, 1998; Precup, 2000; Sutton et al., 1999; Vezhnevets et al., 2017) jointly learn a low-level policy for performing each of the skills together with a…”
Hierarchical Solution of Markov Decision Processes using Macro-actions (1998, background): “The value of a policy (cid:25) can be shown to satisfy [11] V(cid:25)(s) = R(s;(cid:25)(s)) + (cid:12) t2S T(s;(cid:25)(s);t) (cid:1)V(cid:25)(t) A policy (cid:25) is optimal if V(cid:25)(s) (s) for all s 2 S and policies 0 .”