Recent Advances in Hierarchical Reinforcement Learning

By Andrew Barto, Sridhar Mahadevan. 2003

1,440 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “While conventionally these challenges have been approached from the perspective of planning (e.g., TAMP [1]) or hierarchical learning (e.g., HRL [2]), effective high-level reasoning about complex tasks also requires semantic knowledge and understanding of the world.”
Deep learning in neural networks: An overview (2014, methodology): “Numerous alternative HRL techniques have been proposed (e.g., Ring, 1991, 1994; Jameson, 1991; Tenenberg et al., 1993; Moore and Atkeson, 1995; Precup et al., 1998; Dietterich, 2000b; Menache et al., 2002; Doya et al., 2002; Ghavamzadeh and Mahadevan, 2003; Barto and Mahadevan, 2003; Samejima et al.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Readers are directed towards a complete and compact review on advances in hierarchical reinforcement learning [65].”
Reinforcement Learning in Healthcare: A Survey (2019, neutral): “Hierarchical RL (HRL) [64] formalizes hierarchical methods that use abstract states or actions over a hierarchy of subtasks to decompose the original problem, potentially reducing its computational complexity.”
Reinforcement learning in robotics: A survey (2013, background): “It offers a larger set of applicable tools ranging from adaptive discretizations (Buşoniu et al., 2010) and function approximation approaches (Sutton and Barto, 1998) to macro-actions or options (Barto and Mahadevan, 2003; Hart and Grupen, 2011).”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “In the context of hierarchical reinforcement learning [2], Sutton et al.”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “In reinforcement learning, hierarchical approaches aim to construct multiple levels of temporally extended subpolicies [3].”
Learning to Act by Predicting the Future (2016, background): “Second, signiﬁcant progress in behavioral sophistication will likely require temporal abstraction and hierarchical organization of learned skills (Barto & Mahadevan, 2003; Kulkarni et al., 2016a).”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Hierarchical reinforcement learning (HRL) [4] has been proposed as a potential solution that should scale to challenging long-horizon problems, by explicitly introducing temporal abstraction.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “It would be interesting to try hierarchical RL techniques (Barto & Mahadevan (2003)) here.”
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010, methodology): “The Hierarchical Reinforcement Learning [Barto and Mahadevan, 2003] field models repeated decision making by structuring the policy into tasks (actions) composed of subtasks that extend through time (temporal abstraction) and are specific to a subset of the total world state space (state abstraction”
Successor Features for Transfer in Reinforcement Learning (2016, background): “This illustrates how the combination of SFs and GPI can give rise to flexible agents able to perform well in any task of a set of tasks with shared dynamics—which in turn can be seen as both a form of temporal abstraction and a step towards more general hierarchical RL [22, 1].”
Deep Successor Reinforcement Learning (2016, background): “Hierarchical reinforcement learning algorithms [1] such as the options framework [38, 39] provide a ﬂexible framework to create temporal abstractions, which will enable exploration at different time-scales.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “A parallel line of work runs through the hierarchical reinforcement learning (HRL) literature [9], with recent work [8, 27, 50, 58, 84] attempting to learn both lower-level policies as well as a higher-level policy that calls them.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “The line of work involving sequences of instructions has strong ties to Hierarchical RL [Barto and Mahadevan, 2003], with individual sentences or clauses from instructions corresponding to subtasks [ Branavan et al. , 2010 ] .”
Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, background): “The second baseline, latent variable, extends the linear action-value function Q(s, a) of the game only baseline with a set of latent variables – i.e., it is a four layer neural network, where the second layer’s units are activated only based on game information.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Numerous alternative HRL techniques have been proposed [204, 206, 117, 279, 295, 171, 195, 50, 162, 51, 15, 215, 11, 260].”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Hierarchical reinforcement learning (HRL), in which multiple layers of policies are trained to perform decision-making and control at successively higher levels of temporal and behavioral abstraction, has long held the promise to learn such difficult tasks [7, 29, 39, 4].”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “Also defining sub-tasks and sub-goals to guide the agent is a well-established principle in hierarchical reinforcement learning (Barto and Mahadevan, 2003).”