Learning Options in Reinforcement Learning

By Martin Stolle, Doina Precup. 2002

391 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Value Prediction Network (2017, background): “Let xt be the observation or a history of observations for partially observable MDPs (henceforth referred to as just observation) and let ot be the option [33, 31, 25] at time t.”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “Under these definitions the optimal skill value function is given by the following equation (Stolle and Precup 2002):”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “While some works (Bacon et al., 2017; Stolle & Precup, 2002; Peng et al., 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhn-evets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills…”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “Building on the framework of options, temporally-extended actions, and hierarchical single-agent RL [8, 26, 33, 38], early work on hier-”
The Option-Critic Architecture (2016, background): “Discovering temporal abstractions autonomously has been the subject of extensive research efforts in the last 15 years (McGovern and Barto 2001; Stolle and Precup 2002; Menache, Mannor, and Shimkin 2002; Şimşek and Barto 2009; Silver and Ciosek 2012), but approaches that can be used naturally with c”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Each subpolicy π is uniquely associated with a symbol b implemented as a neural network that maps from a state s i to distributions over A + , and chooses an action a i by sampling from this distribution.”
Variational Intrinsic Control (2016, background): “…to option learning where the goal is to ﬁnd a small number of options that are useful for a particular task (Sutton et al., 1999; McGovern & Barto, 2001; Stolle & Precup, 2002; Silver & Ciosek, 2012; Kulkarni et al., 2016; Mankowitz et al., 2014; Vezhnevets et al., 2016; Bacon & Precup, 2015).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “This ranges from the classic option-critic architecture (Sutton et al., 1999; Stolle & Precup, 2002; Perkins et al., 1999) to some of the more recent work (Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018; Hausman et al., 2018).”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Classically, the work on HRL focused on discrete state domains, where state visitation and transition statistics can be used to construct heuristic sub-goals for low-level policies [37, 27, 5].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “A popular approach there is to use statistics about state transitions to identify bottleneck states (Stolle & Precup, 2002; Mannor et al., 2004; S¸ims¸ek et al., 2005).”
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning (2017, background): “An agent that can take temporally extended actions represented as action primitives or skills can more easily explore the environment (Stolle & Precup, 2002).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Classically, the work on HRL has focused on learning only the high-level policy given a set of hand-engineered low-level policies [54, 33, 8], or more generically options policies with ﬂexible termination conditions [57, 46].”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, background): “A bottleneck state is deﬁned as one that lies on a wide variety of rewarding trajectories (McGovern and Barto, 2001; Stolle and Precup, 2002) or one that otherwise serves to separate regions of state space in a graph-theoretic sense (Menache et al., 2002; ¸Sim¸sek et al., 2005; Kazemitabar and…”
Reinforcement learning in the brain (2009, background): “One question that frequently arises in these theoretical studies is, where do these previously learnedmodules come from, or, more importantly, how does an agent learn useful modules (e.g. Simsek and Barto (2004) and Stolle and Precup (2002)).”
Automatic Construction of Temporally Extended Actions for MDPs Using Bisimulation Metrics (2011, methodology): “Subgoals can be identified based on a learned model of the environment, using graph theoretic techniques [Mannor et al., 2004; Šimšek et al., 2005], or based on trajectories through the system, without learning a model [McGovern and Barto, 2001; Stolle and Precup, 2002].”
Skill Characterization Based on Betweenness (2008, methodology): “More importantly, our characterization serves as a useful guide for developing incremental skill-discovery algorithms that do not rely on knowing or representing the interaction graph in its entirety.”
Learning Abstract Options (2018, background): “Discovering these temporal abstractions autonomously has been the subject of extensive research [16, 28, 17, 27, 26] with approaches that can be used in continuous state and/or action spaces only recently becoming feasible [9, 20, 15, 14, 10, 31, 3].”
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning (2019, background): “Recently, many researchers have focused on learning hierarchical policies [3, 44, 47] in simple domains.”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, neutral): “, “go forward until wall”, “climb ladder up”, “find the key”, “use key to open door”) [72 ,73,74].”