Dynamic abstraction in reinforcement learning via clustering

By Shie Mannor, Ishai Menache, Amit Hoze, Uri Klein. 2004

262 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Industrial and Manufacturing Engineering

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Graying the black box: Understanding DQNs (2016, methodology): “Such methods decompose the learning task into simpler subtasks using graph partitioning (Menache et al., 2002; Mannor et al., 2004; Şimşek et al., 2005) and path processing mechanisms (Stolle, 2004; Thrun, 1998).”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, positive): “ represents optimal behavior towards the goal g. Our work is inspired by these papers and builds upon them. There has also been a lot of work on option discovery in the tabular value function setting [26, 38, 25, 27]. In more recent work, Machado et al. [24] presented an option discovery algorithm w”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Abstraction and Bisimulation Traditionally, representation learning has been framed as learning or identifying abstractions of the state or action space of an environment (Andre and Russell, 2002; Mannor et al., 2004; Dearden and Boutilier, 1997; Abel et al., 2018).”
Deep Successor Reinforcement Learning (2016, background): “There has also been a lot of work on option discovery in the tabular setting [14, 18–20 , 34 Our model is also related to the literature on value function approximation using deep neural networks.”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Classically, the work on HRL focused on discrete state domains, where state visitation and transition statistics can be used to construct heuristic sub-goals for low-level policies [37, 27, 5].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “A popular approach there is to use statistics about state transitions to identify bottleneck states (Stolle & Precup, 2002; Mannor et al., 2004; S¸ims¸ek et al., 2005).”
Learning Plannable Representations with Causal InfoGAN (2018, background): “Methods such as in [27, 38] exploit the value function for measuring state similarity, and are therefore limited to the task defined by the reward.”
Scaling Up Approximate Value Iteration with Options: Better Policies with Fewer Iterations (2014, background): “In addition, much effort has gone into algorithms that learn “good” options for exploration (Stolle & Precup, 2002; Mannor et al., 2004).”
Automatic Construction of Temporally Extended Actions for MDPs Using Bisimulation Metrics (2011, methodology): “Subgoals can be identified based on a learned model of the environment, using graph theoretic techniques [Mannor et al., 2004; Šimšek et al., 2005], or based on trajectories through the system, without learning a model [McGovern and Barto, 2001; Stolle and Precup, 2002].”
Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes (2015, background): “, 2005], clustering [Mannor et al., 2004], bottleneck state discovery [McGovern and Barto, 2001; Menache et al.”
Options of Interest: Temporal Abstraction with Interest Functions (2020, methodology): “While we use rewards in our gradient-based algorithm, our qualitative analysis also indicates some clustering of states in which a given option starts, as in (Lakshminarayanan et al. 2016; Bacon 2013; Mannor et al. 2004).”
Skill Characterization Based on Betweenness (2008, methodology): “More importantly, our characterization serves as a useful guide for developing incremental skill-discovery algorithms that do not rely on knowing or representing the interaction graph in its entirety.”
In search of intelligence: evolving a developmental neuron capable of learning (2014, methodology): “The approach is inspired by the ‘clustering approach’ used to improve learning skills of an agent (Mannor, Menache, Hoze, & Klein, 2004).”
Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective (2008, methodology): “Graph partitioning methods are then used to identify states that constitute bottlenecks or access points within the graph, which are then designated as option subgoals ( Mannor, Menache, Hoze, & Klein, 2004; Menache, Mannor, & Shimkin, 2002; Simsek, Wolfe, & Barto, 2005; see also Hengst, 2002; Jons”
Identifying useful subgoals in reinforcement learning by local graph partitioning (2005, background): “The utility of access states as subgoals has been argued previously in the literature [9, 10, 11, 12].”
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (2009, methodology): “Graph-based methods [9, 10, 11] build a state graph and use its properties (e.”
A Laplacian Framework for Option Discovery in Reinforcement Learning (2017, background): “, 2005; Şimşek & Barto, 2008) and clustering of states (Mannor et al., 2004; Bacon, 2013; Lakshminarayanan et al., 2016).”
Hierarchical reinforcement learning for efficient exploration and transfer (2020, background): “The ability to compress the state space is also critical to our work (Mannor et al., 2004; Vezhnevets et al., 2017).”