Linear options

By Jonathan Sorg, Satinder Singh. 2010

37 citations recorded by the source corpus.

Learning, planning, and representing knowledge in large state spaces at multiple levels of temporal abstraction are key, long-standing challenges for building flexible autonomous agents. The options framework provides a formal mechanism for specifying and learning temporally-extended skills. Although past work has demonstrated the benefit of acting according to options in continuous state spaces, one of the central advantages of temporal abstraction---the ability to plan using a temporally abstract model---remains a challenging problem when the number of environment states is large or infinite. In this work, we develop a knowledge construct, the linear option, which is capable of modeling temporally abstract dynamics in continuous state spaces. We show that planning with a linear expectation model of an option's dynamics converges to a fixed point with low Temporal Difference (TD) error. Next, building on recent work on linear feature selection, we show conditions under which a linear feature set is sufficient for accurately representing the value function of an option policy. We extend this result to show conditions under which multiple options may be repeatedly composed to create new options with accurate linear models. Finally, we demonstrate linear option learning and planning algorithms in a simulated robot environment.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “Recently, several methods have been proposed to learn options in real-time by using varying reward functions [35] or by composing existing options [28].”
The Option-Critic Architecture (2016, methodology): “Earlier work on linear options (Sorg and Singh 2010) also used compositionality to plan using linear expectation models for options.”
Universal Option Models (2014, methodology): “Options are often used to specify action abstraction [Precup, 2000, Sorg and Singh, 2010, Sutton et al., 1999].”
Iterative Hierarchical Optimization for Misspecified Problems (IHOMP) (2016, methodology): “In our experiments, we used a straightforward variant of LSTD [Sorg and Singh, 2010].”
From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning (2018, background): “However, although options can speed up planning by shortening the effective search depth (Sorg & Singh, 2010; Powley, Whitehouse, & Cowling, 2012), SMDP-style planning in the kinds of large, continuous state spaces that characterize robot problems is still very difficult.”
Approximate Value Iteration with Temporally Extended Actions (2015, background): “In large-or continuous-state MDPs, the linear options framework enables the construction of option models by composing models for primitive actions (Sorg & Singh, 2010).”
Symmetry Learning for Function Approximation in Reinforcement Learning (2017, methodology): “Recent approaches for incorporating TAA use linear function approximation [17, 18] and have been shown to be effective.”