Using relative novelty to identify useful temporal abstractions in reinforcement learning

By Özgür Simsek, Andrew Barto. 2004

258 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, background): “mination conditions, for instance, might be determined by identifying possibly useful subgoal states [47, 14, 16, 38] that an agent may want to reach.”
The strategic student approach for life-long exploration and learning (2012, background): “Heuristics to start such process include relevant point detection [25], bottlenecks [56], [57] among others.”
Arousal regulation and affective adaptation to human responsiveness by a robot that explores and learns a novel environment (2014, methodology): “To that end, their model includes a two-dimensional vector of pleasure and arousal following the circumplex model of emotions introduced by Russel (1980).”
Hierarchical Approaches (2012, background): “Şimşek and Barto (2004) use a relative novelty measure to identify sub-goal states.”
Value Preserving State-Action Abstractions (2020, background): “Options are known to aid in transfer (Konidaris and Barto, 2007, 2009; Brunskill and Li, 2014; Topin et al., 2015), encourage better exploration (Şimşek and Barto, 2004, 2009; Brunskill and Li, 2014; Bacon et al., 2017; Fruit and Lazaric, 2017; Machado et al., 2017; Tiwari and Thomas, 2019;…”
Approximate Value Iteration with Temporally Extended Actions (2015, methodology): “Here we assume that the landmark states are given, but they could be acquired through analyzing the dynamics of M̂ as in the work of Simsek and Barto (2004) or from demonstrations as in the work of Konidaris, Kuindersma, Barto, and Grupen (2010).”
Discovering Options for Exploration by Minimizing Cover Time (2019, methodology): “S¸ims¸ek & Barto (2009) generalized the concept of bottleneck states to the (shortest-path) betweenness of the graph to capture how pivotal the state is.”
Reinforcement learning in the brain (2009, background): “One question that frequently arises in these theoretical studies is, where do these previously learnedmodules come from, or, more importantly, how does an agent learn useful modules (e.g. Simsek and Barto (2004) and Stolle and Precup (2002)).”
Skill Characterization Based on Betweenness (2008, methodology): “Our skill definition captures the “bottleneck” concept, which has inspired many of the existing skill discovery algorithms [3, 6, 4, 5, 7, 8, 9].”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, methodology): “In particular, the RPI framework is an iterative approach, which interleaves the two phases of generating basis functions by sampling trajectories from policies, and then subsequently finding improved policies from the augmented set of basis functions.”
An intrinsic reward mechanism for efficient exploration (2006, background): “Many existing discovery methods can identify this skill, e.g., Mannor et al. (2004), Şimşek and Barto (2004), Şimşek et al. (2005).”
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (2009, methodology): “In previous research, option goal states have been selected by a variety of methods, the most common relying on computing visit or reward statistics over individual states to identify useful subgoals [5, 6, 7, 8].”
Transfer in Reinforcement Learning via Shared Features (2012, background): “Previous research has selected goal states by a variety of methods, for example: visit frequency and reward gradient (Digney, 1998), visit frequency on successful trajectories (McGovern and Barto, 2001), variable change frequency (Hengst, 2002), relative novelty (Şimşek and Barto, 2004), clustering ”