HQ-Learning

By Marco A. Wiering, J. Urgen Schmidhuber. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “The goal of acquiring primitives for the purpose of enabling efficient learning on downstream tasks is also a central goal of hierarchical reinforcement learning (HRL), which has been a long-standing area of research [11, 51, 32, 43, 12].”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, methodology): “…learning (Sutton et al., 1999; Precup, 2000; Dayan & Hinton, 1993; Dietterich, 2000; Boutilier et al., 1997; Dayan, 1993; Kaelbling, 2014; Parr & Russell, 1998; Precup et al., 1997; 1998; Schmidhuber, 1991; Sutton, 1995; Wiering & Schmidhuber, 1997; Vezhnevets et al., 2016; Bacon et al., 2017).”
First Experiments with PowerPlay (2012, background): “This type of self-modularization is more general than what can be found in traditional (non-inventive) modular reinforcement learning (RL) systems whose action sequences are chunked into macros to be re-used by higher-level macros, like in the options framework [30], or in hierarchical RL [32].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “While HRL frameworks such as Feudal RL [48] and options [275, 16, 261] do not directly address the problem of automatic subgoal discovery, HQ-Learning [309] automatically decomposes problems in partially observable environments into sequences of simpler subtasks that can be solved by memoryless poli”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “The options framework [21, 28] provides a general formulation.”
Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning (2002, background): “, [4], [6], [18]) is to decompose the learning task into simpler subtasks, which is a natural procedure also performed by humans.”
Quantum Reinforcement Learning (2008, methodology): “Temporal abstraction and decomposition methods have been explored to solve such problems as RL and dynamic programming (DP) to speedup learning [11]–[15].”
Inter-module credit assignment in modular reinforcement learning (2003, background): “We discuss the remaining problems and possible future work in section 6.”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, methodology): “apparently being unaware of previous work on hierarchical RL systems that can discover their own subgoals [60], [102], [61], [118], [79], [1].”
Learning Finite-State Controllers for Partially Observable Environments (1999, background): “For instance, some researchers [14, 3, 32] try to learn memoryless (or reactive) policies, McCallum’s learning algorithm [19, 20] uses a finite-horizon memory, Wiering and Schmidhuber’s HQL [31] learns finite sequences of reactive policies using an implicit memory of some of the previous observation”
Intrinsically Motivated Learning of Real-World Sensorimotor Skills with Developmental Constraints (2013, background): “…learning and option theory, and more generally approaches to cumulative learning, and the reader is referred to the following articles for an overview of those techniques (Bakker and Schmidhuber, 2004; Barto et al., 2004; Ring, 1994; Sutton et al., 1999; Wiering and Schmidhuber, 1997).”
From hardware and software to kernels and envelopes: a concept shift for robotics, developmental psychology, and brain sciences (2011, methodology): “…1992; Tani and Nolfi, 1999; Tani, 2007) or the option framework (Sutton et al., 1999) permit hierarchical learning where chunks of behavior can be compiled and continuously adapted to be used later on (see also (Dayan et al., 1993; Ring, 1994; Wiering and Schmidhuber, 1997) for related methods).”
Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning (2000, methodology): “For example, hierarchical Q-learning methods have been used for solving POMDPs by dividing the state space into several regions in which each task is reduced to a Markov decision problem (MDP) [26].”