Intra-Option Learning about Temporally Abstract Actions

By Richard S. Sutton, Doina Precup, Satinder Singh. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “We illustrate this with an experiment on a classic four room grid-world [Sutton et al., 1998].”
First Experiments with PowerPlay (2012, background): “This type of self-modularization is more general than what can be found in traditional (non-inventive) modular reinforcement learning (RL) systems whose action sequences are chunked into macros to be re-used by higher-level macros, like in the options framework [30], or in hierarchical RL [32].”
GQ(lambda): A general gradient algorithm for temporal-difference prediction learning with eligibility traces (2010, background): “Intra-option learning (Sutton, Precup, and Singh, 1998) is seen as a potential method for temporal-abstraction in reinforcement learning.”
Hierarchical Imitation and Reinforcement Learning (2018, background): “Several reinforcement learning approaches to learning hierarchical policies have been explored, foremost among them the options framework (Sutton et al., 1998; 1999; Fruit & Lazaric, 2017).”
Efficient skill learning using abstraction selection (2009, background): “Policy learning is usually by an offpolicy reinforcement learning algorithm so that the agent can update many options simultaneously after taking an action [Sutton et al., 1998].”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “(4) and recognizing qt = E [∑H j=t νt+1:jrj | st, at ] , we obtain a policy gradient in terms of the q-function (Sutton et al., 1998), Z(θ) = E [∑H t=0 ν0:tgtqt ] .”
Hierarchical Relative Entropy Policy Search (2014, background): “In this case, the data efficiency for learning the options can be significantly improved by leveraging all information collected during the execution of one option, also denoted as intra-option learning (Sutton et al., 1998).”
Probabilistic policy reuse in a reinforcement learning agent (2006, background): “Options can also be used in SMDPs by learning their behavior on line [13].”
Building Portable Options: Skill Transfer in Reinforcement Learning (2007, background): “Policy learning is usually performed by an off-policy reinforcement learning algorithm so that the agent can update many option simultaneously after taking an action [Sutton et al., 1998].”
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information (2018, methodology): “Under the options framework (Sutton et al., 1998), an option, indexed by o ∈ O consists of a sub-policy π(a|s, o), a termination policy π(b|s, ō) and an option activation policy π(o|s).”
A Bayesian Nonparametic Approach for Estimating Individualized Treatment-Response Curves (2016, background): “A related problem in reinforcement learning is off-policy evaluation where the goal is to estimate the value of a policy (sequence of actions) from data collected by another policy (Sutton et al., 1998).”
Hierarchical Memory-Based Reinforcement Learning (2000, unknown): “Previous work has addressed this issue by breaking down a large task into a hierarchy of subtasks or abstract behaviors [1, 3, 5].”
AUTOMATED DISCOVERY OF OPTIONS IN REINFORCEMENT LEARNING (2003, methodology): “When learning with options, Intra Option Q-Learning was used [29][22].”
Robot learning via socially guided exploration (2007, background): “Each T ∈ Tasks can learn and expand from every experience (also referred to as intra-option learning [19]).”
Eligibility Traces for Off-Policy Policy Evaluation (2000, background): “Off-policy learning enables the agent to use its experience to learn about the values and models of all the policies in parallel, even though it can follow only one policy at a time (Sutton, Precup & Singh, 1998).”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “We can approximate this value directly with Monte-Carlo methods, that is, take the average of the accumulated reward computed for several roll-outs starting from this position (Sutton and Barto, 1998, Chapter 5).”