Compositional Planning Using Optimal Option Models

By David Silver, Kamil Andrzej Ciosek. 2012

77 citations recorded by the source corpus.

In this paper we introduce a framework for option model composition. Option models are temporal abstractions that, like macro-operators in classical planning, jump directly from a start state to an end state. Prior work has focused on constructing option models from primitive actions, by intra-option model learning; or on using option models to construct a value function, by inter-option planning. We present a unified view of intra- and inter-option model learning, based on a major generalisation of the Bellman equation. Our fundamental operation is the recursive composition of option models into other option models. This key idea enables compositional planning over many levels of abstraction. We illustrate our framework using a dynamic programming algorithm that simultaneously constructs optimal option models for multiple subgoals, and also searches over those option models to provide rapid progress towards other subgoals.

Subfields: Artificial Intelligence, Control and Systems Engineering, Ocean Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Ocean Engineering

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “A variety of reinforcement learning architectures have focused on learning temporal abstractions, such as options (Sutton et al., 1999b), with policies that may maximise pseudo-rewards (Konidaris & Barreto, 2009; Silver & Ciosek, 2012).”
Variational Intrinsic Control (2016, background): “…to option learning where the goal is to ﬁnd a small number of options that are useful for a particular task (Sutton et al., 1999; McGovern & Barto, 2001; Stolle & Precup, 2002; Silver & Ciosek, 2012; Kulkarni et al., 2016; Mankowitz et al., 2014; Vezhnevets et al., 2016; Bacon & Precup, 2015).”
Combating the Compounding-Error Problem with a Multi-step Model (2019, background): “Similarly, option models are multi-step models that are conditioned on one speciﬁc policy and a termination condition [Precup and Sutton, 1998; Sutton et al. , 1999; Silver and Ciosek, 2012].”
Scaling Up Approximate Value Iteration with Options: Better Policies with Fewer Iterations (2014, background): “Silver & Ciosek (2012) demonstrate impressive results for planning while simultaneously composing new options in multiple discrete state MDPs. Silver & Ciosek further show that the options generated by their proposed algorithm converge to optimal, but their theoretical analysis does not compare…”
From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning (2018, background): “However, while the use of high-level skills can significantly improve performance (Sutton, Precup, & Singh, 1999; Silver & Ciosek, 2012), planning in the high-dimensional, continuous state spaces that characterize robotics problems (and many other applications) remains very difficult.”
Model-based hierarchical reinforcement learning and human action control (2014, background): “The ability to plan hierarchically can have a dramatic impact on planning performance [16,17,19].”
Approximate Value Iteration with Temporally Extended Actions (2015, methodology): “However, they may also play an important role in planning (Precup & Sutton, 1997; Precup et al., 1998; Sutton et al., 1999; Silver & Ciosek, 2012).”
Learning Abstract Options (2018, background): “Compositional planning where options select other options was first considered in [26].”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “Contemporary deep reinforcement learning models have a ways to go to achieve robust generalization [Nichol et al., 2018], efﬁcient planning over ﬂexible timescales [Silver and Ciosek, 2012], and long-term credit assignment [Osband et al., 2019].”
Time-regularized interrupting options (2014, background): “However, planning with temporally extended options can lead to significantly faster convergence (Mann & Mannor, 2014; Precup & Sutton, 1997; Sutton et al., 1999; Silver & Ciosek, 2012).”
Planning with macro-actions in decentralized POMDPs (2014, background): “In single-agent domains, hierarchical approaches [4] using higher-level, temporally extended macro-actions have been explored as a more natural way to represent and solve problems, leading to significant performance improvements in planning [26, 30].”