Discovering Hierarchy in Reinforcement Learning with HEXQ

By Bernhard Hengst. 2002

246 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “f research (e.g. Daniel et al. (2012)) investigates learning hierarchical policies without any supervision, such hierarchies are empirically difﬁcult to learn directly from unconstrained interaction (Hengst, 2002). The bulk of existing work instead relies on additional information (in the form of in”
Meta-Learning: A Survey (2018, neutral): “, 2006; Bakker and Heskes, 2003), Bayesian networks (Niculescu-Mizil and Caruana, 2005), clustering (Thrun, 1998) and reinforcement learning (Hengst, 2002; Dietterich et al., 2002).”
Using relative novelty to identify useful temporal abstractions in reinforcement learning (2004, methodology): “This work is closely related to a number of methods proposed earlier in the literature, most notably to Menache et al. (2002), Hengst (2002), McGovern (2002), and McGovern and Barto (2001), all of which identify subgoal states and define temporally-extended activities that take the agent…”
Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective (2008, methodology): “…methods are then used to identify states that constitute bottlenecks or access points within the graph, which are then designated as option subgoals (Mannor, Menache, Hoze, & Klein, 2004; Menache, Mannor, & Shimkin, 2002; Simsek, Wolfe, & Barto, 2005; see also Hengst, 2002; Jonsson & Barto, 2005).”
Identifying useful subgoals in reinforcement learning by local graph partitioning (2005, background): “In addition, Hengst (2002) has used the notion of a subgoal in performing temporal and spatial abstraction simultaneously, defining subgoals to be those states that lead to transitions that the agent can not correctly represent or predict at the current level of state abstraction.”
Probabilistic policy reuse in a reinforcement learning agent (2006, background): “Hierarchical RL uses different abstraction levels to organize subtasks [3], and some approaches are able to learn such a hierarchy [6].”
Recent Advances in Hierarchical Reinforcement Learning (2003, methodology): “Another approach called HEXQ was recently proposed by Hengst [24].”
Object focused q-learning for autonomous agents (2013, unknown): “Some of these approaches rely on a hierarchical domain-specific structure that has to be specified by a system designer [2] or derived from heuristics that apply only to restricted classes of domains [8] [15] [17].”
Tree based hierarchical reinforcement learning (2002, background): “Hengst (2000) has recently generated hierarchies of a limited form by finding a fixed ordering of the state variables. While this approach is effective, it is limited by requiring the same variable ordering in all regions of the state space. The second main style (Sutton et al., 1998; Precup, 2000) ”
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (2009, background): “In domains with factored state spaces, the agent may create options to change infrequently changing variables [12, 13].”
Learning Shared Representations in Multi-task Reinforcement Learning (2016, background): “s (Konidaris & Barto,2007), macro-actions literature (Dietterich,2000) and are hve the potential to drastically improve the efﬁciency and scalability of RL methods (Barto & Mahadevan ,2003), (Hengst 2002). In the Learning share representations for value functions in Multi-Task RL following w”
Online Planning for Large Markov Decision Processes with Hierarchical Decomposition (2015, background): “…promising approach that has been drawing much research interest is discovering the hierarchical structure automatically from state-action histories in the environment, either online or ofﬂine [Hengst 2002; Stolle 2004; Bakker and Schmidhuber 2004; S¸ims¸ek et al. 2005; Mehta et al. 2008, 2011].”
An intrinsic reward mechanism for efficient exploration (2006, methodology): “…methods proceed by generating a reward function that the skill should maximize, typically by identifying a set of states that are useful to reach and defining a skill reward function whose optimal policy efficiently takes the agent to these states, e.g., Hengst (2002), McGovern and Barto (2001).”
Transfer in Reinforcement Learning via Shared Features (2012, background): “Other research has focused on extracting options by exploiting commonalities in collections of policies over a single state space (Thrun and Schwartz, 1995; Bernstein, 1999; Perkins and Precup, 1999; Pickett and Barto, 2002).”
An algebraic approach to abstraction in reinforcement learning (2004, background): “Autonomously discovering hierarchical structure is an active topic of research recently (Digney, 1998; McGovern and Barto, 2001; Hengst, 2002; Pickett and Barto, 2002).”