Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization

By Bram Bakker. 2005

151 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “Measures of interestingness based on a measure of competence to perform a skill were studied in [93], as well as in [60] where a selector chooses to perform different skills depending on the temporal difference error to reach each skill.”
Hindsight Experience Replay (2017, background): “Learning simultaneously to perform multiple tasks has been also investigated for a long time in the context of Hierarchical Reinforcement Learning, e.g. Bakker and Schmidhuber (2004); Vezhnevets et al. (2017).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Whenever the STOP action is sampled, control advances to the next subpolicy in the sketch. exactly equal to the state-value function V π ( s i ) = E π q i for the target policy π starting in state s i .”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Numerous alternative HRL techniques have been proposed [204, 206, 117, 279, 295, 171, 195, 50, 162, 51, 15, 215, 11, 260].”
Neural Modular Control for Embodied Question Answering (2018, background): “Relative to other work on automatic subgoal discovery in hierarchical reinforcement learning [12–14], we show that given knowledge of the problem structure, simple heuristics are quite effective in breaking down long-range planning into sequential subgoals.”
Intrinsically Motivated Learning of Real-World Sensorimotor Skills with Developmental Constraints (2013, methodology): “…learning and option theory, and more generally approaches to cumulative learning, and the reader is referred to the following articles for an overview of those techniques (Bakker and Schmidhuber, 2004; Barto et al., 2004; Ring, 1994; Sutton et al., 1999; Wiering and Schmidhuber, 1997).”
Online Planning for Large Markov Decision Processes with Hierarchical Decomposition (2015, background): “…promising approach that has been drawing much research interest is discovering the hierarchical structure automatically from state-action histories in the environment, either online or ofﬂine [Hengst 2002; Stolle 2004; Bakker and Schmidhuber 2004; S¸ims¸ek et al. 2005; Mehta et al. 2008, 2011].”
Autonomous identification, categorization and generalization of policies based on task type (2011, background): “This is addressed in hierarchical learning techniques [1][2][3][4][7].”
Hierarchical Approaches (2012, background): “Bakker and Schmidhuber (2004)’s HASSLE algorithm discovers subgoals by learning how to transition between abstract states.”
GRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning (2016, background): “Some efforts have been made toward this direction in the field of hierarchical reinforcement learning [13] but some of these works [2], [41], [48] focus on searching for subgoals on the basis of some sort of externally given tasks.”
Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. (2016, methodology): “Related uses of intrinsic motivation with a hierarchical reinforcement learning framework were demonstrated in (Bakker and Schmidhuber, 2004; Kulkarni et al., 2016).”
Toward Good Abstractions for Lifelong Learning (2017, result): “The results support the (well established) conclusion that effective hierarchies reduce learning difficulty [9, 14, 3, 2, 7, 16].”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, methodology): “apparently being unaware of previous work on hierarchical R L systems that can discover their own subgoals [60], [102], [6 1], [118], [79], [1].”
Near Optimal Behavior via Approximate State Abstraction (2016, methodology): “ving abstractions that maintain optimality. Dietterich [12] developed MAXQ, a framework for composing tasks into an abstracted hierarchy where state aggregation can be applied. Bakker and Schmidhuber [4] also target hierarchical abstraction, focusing on subgoal discovery. Jong and Stone [21] introdu”
Maximizing Fun by Creating Data with Easily Reducible Subjective Complexity (2013, background): “Some goal-seeking RL systems search a space of possible subgoal combinations, internally rewarding subsystems whose policies learn to achieve those subgoals (Bakker and Schmidhuber 2004; Ring 1994; Schmidhuber and Wahnsiedler 1992; Wiering and Schmidhuber 1998a).”