Temporal abstraction in reinforcement learning

By Doina Precup, Richard S. Sutton. 2000

383 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, background): “One way in which MDPs can be extended is via the Semi-Markov Decision Processes (SMDP) framework [54].”
Value Prediction Network (2017, background): “Let xt be the observation or a history of observations for partially observable MDPs (henceforth referred to as just observation) and let ot be the option [33, 31, 25] at time t.”
Independently Controllable Features (2017, background): “There are many more works that go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al., 2016).”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “This differs from (and is somewhat easier than) the setting considered in some previous work, often known as off-policy policy evaluation, which aims to estimate the whole value function (Precup et al., 2000; 2001; Sutton et al., 2015).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Recent work has investigated deep hierarchical agents [1, 48, 16, 31], building on methods that explicitly design a hierarchy over actions [32, 12], or utilize options [43, 35].”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “Most importantly, off-policy methods allow an agent to learn about many different policies at once, forming the basis for a predictive understanding of an agent’s environment [Sutton et al., 2011, White, 2015] and enabling the learning of options [Sutton et al., 1999, Precup, 2000].”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “Building on the framework of options, temporally-extended actions, and hierarchical single-agent RL [8, 26, 33, 38], early work on hier-”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, background): “Building hierarchical agents is a long standing topic in re-inforcement learning (Sutton et al., 1999; Precup, 2000; Dayan & Hinton, 1993; Dietterich, 2000; Boutilier et al., 1997; Dayan, 1993; Kaelbling, 2014; Parr & Russell, 1998; Precup et al., 1997; 1998; Schmidhuber, 1991; Sutton, 1995;…”
The Option-Critic Architecture (2016, background): “In reinforcement learning, options (Sutton, Precup, and Singh 1999; Precup 2000) provide a framework for deﬁn-ing such courses of action and for seamlessly learning and planning with them.”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Once a collection of high-level actions exists, agents are faced with the problem of learning meta-level (typically semi-Markov) policies that invoke appropriate high-level actions in sequence (Precup, 2000).”
Disentangling the independently controllable factors of variation by interacting with the world (2018, neutral): “The idea that learning good representations is vital for solving most kinds of real-world problems is not new, both in the supervised learning literature [Bengio, 2009, Goodfellow et al., 2016], and in the RL literature [Dayan, 1993, Precup, 2000].”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “The options framework [39, 32], a popular formulation for HRL, proposes a termination policy for each sub-policy (option).”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “The options framework [21, 28] provides a general formulation.”
Independently Controllable Factors (2017, methodology): “, 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Classically, the work on HRL has focused on learning only the high-level policy given a set of hand-engineered low-level policies [54, 33, 8], or more generically options policies with ﬂexible termination conditions [57, 46].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “One class of methods (Bacon et al., 2017; Frans et al., 2017; Kaelbling, 1993a; Kulkarni et al., 2016; Nachum et al., 2018; Parr and Russell, 1998; Precup, 2000; Sutton et al., 1999; Vezhnevets et al., 2017) jointly learn a low-level policy for performing each of the skills together with a…”
From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning (2018, background): “Options are often—especially in research on skill discovery methods—constructed to reach a subgoal (Precup, 2000).”
Skill Characterization Based on Betweenness (2008, methodology): “We represented skills using the options framework [14, 15].”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, methodology): “The options framework [31, 26] provides a canonical formalisation of a two-level HRL algorithm.”