Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning

By Ishai Menache, Shie Mannor, Nahum Shimkin. 2002

306 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “). Numerous alternative HRL techniques have been proposed (e.g., Ring, 1991, 1994; Jameson, 1991; Tenenberg et al., 1993; Weiss, 1994; Moore and Atkeson, 1995; Precup et al., 1998; Dietterich, 2000b; Menache et al., 2002; Doya et al., 2002; Ghavamzadeh and Mahadevan, 2003; Barto and Mahadevan, 2003;”
Graying the black box: Understanding DQNs (2016, methodology): “Such methods decompose the learning task into simpler subtasks using graph partitioning (Menache et al., 2002; Mannor et al., 2004; Şimşek et al., 2005) and path processing mechanisms (Stolle, 2004; Thrun, 1998).”
The Option-Critic Architecture (2016, background): “…abstractions autonomously has been the subject of extensive research efforts in the last 15 years (McGovern and Barto 2001; Stolle and Precup 2002; Menache, Mannor, and Shimkin 2002; S¸ims¸ek and Barto 2009; Silver and Ciosek 2012), but approaches that can be used naturally with continuous state…”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Techniques for learning options with less guidance than the present work include Bacon & Precup (2015) and Vezhn-evets et al. (2016), and other general hierarchical policy learners include Daniel et al. (2012), Bakker & Schmidhu-ber (2004) and Menache et al. (2002).”
Deep Successor Reinforcement Learning (2016, background): “There has also been a lot of work on option discovery in the tabular setting [14, 18–20 , 34 Our model is also related to the literature on value function approximation using deep neural networks.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Numerous alternative HRL techniques have been proposed [204, 206, 117, 279, 295, 171, 195, 50, 162, 51, 15, 215, 11, 260].”
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information (2018, methodology): “Many different approaches such as multiple-instance learning (McGovern & Barto, 2001), graph based algorithms (Menache et al., 2002; Şimşek et al., 2005) have been used to find such bottleneck states.”
Skill Characterization Based on Betweenness (2008, background): “Our characterization captures and generalizes (at least intuitively) the bottleneck concept, which has inspired many of the existing skill-discovery algorithms [3, 4, 5, 6, 7, 8, 9].”
A Laplacian Framework for Option Discovery in Reinforcement Learning (2017, methodology): “There are many approaches based on this principle, such as methods that use the observed rewards to generate intrinsic rewards leading to new value functions (e.g., McGovern & Barto, 2001; Menache et al., 2002; Konidaris & Barto, 2009), methods that use the observed rewards to climb a gradient (e.”
Multi-Level Discovery of Deep Options (2017, background): “Naively, the number of different contexts that the d highest levels can provide to lower levels grows exponentially with d .”
Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes (2015, background): “Local graph partitioning [Şimşek et al., 2005], clustering [Mannor et al., 2004], bottleneck state discovery [McGovern and Barto, 2001; Menache et al., 2002], and betweenness centrality [Simsek and Barto, 2007] all provide graph-theoretic approaches to generating options autonomously.”
Multitask Soft Option Learning (2019, methodology): “Many methods exist for constructing and/or learning skills for particular tasks (Dayan & Hinton, 1993; Sutton et al., 1999; McGovern & Barto, 2001; Menache et al., 2002; Şimşek & Barto, 2009; Gregor et al., 2016; Kulkarni et al., 2016; Bacon et al., 2017; Nachum et al., 2018a).”
Design Principles of the Hippocampal Cognitive Map (2014, background): “A number of authors have used graph partitioning techniques to discover subgoals [30, 39].”
Divide and Conquer: Hierarchical Reinforcement Learning and Task Decomposition in Humans (2013, background): “S ¸im¸sek (2008) and S ¸im¸sek and Barto (2009) proposed that option discovery might be fruitfully accomplished by identifying states at local maxima of graph betweenness (for related ideas, see also S ¸im¸sek et al. (2005); Hengst (2002); Jonsson and Barto (2006); Menache et al. (2002).”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, background): “e that lies on a wide variety of rewarding trajectories (McGovern and Barto, 2001; Stolle and Precup, 2002) or one that otherwise serves to separate regions of state space in a graph-theoretic sense (Menache et al., 2002; Sim¸ sek¸ et al., 2005; Kazemitabar and Beigy, 2009; Machado et al., 2017). Th”
Optimal Policies Tend To Seek Power (2019, background): “Menache et al. [2002] identify and navigate towards convergently instrumental bottleneck states.”
Using relative novelty to identify useful temporal abstractions in reinforcement learning (2004, methodology): “This work is closely related to a number of methods proposed earlier in the literature, most notably to Menache et al. (2002), Hengst (2002), McGovern (2002), and McGovern and Barto (2001), all of which identify subgoal states and define temporally-extended activities that take the agent…”