Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

By Richard S. Sutton, Doina Precup, Satinder Singh. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Investigating Contingency Awareness Using Atari 2600 Games (2012, background): “Solving these harder games may require more elaborate reinforcement learning techniques, such as smarter exploration (Brafman and Tennenholtz 2003), temporally extended actions (Sutton, Precup, and Singh 1999) or Monte- Carlo Tree Search (Kocsis and Szepesvári 2006).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “In addition to learning a reward function, we could also learn side-constraints for low-level or high-level actions ( options ; Sutton et al., 1999) to prevent unacceptable outcomes.”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “The model is typically conditioned on the selected action, or a temporally abstract behavior such as an option [43].”
Learning Parameterized Skills (2012, methodology): “The most widely used hierarchical reinforcement learning formalism is the options framework (Sutton et al., 1999), where high-level options (also called skills) define temporally extended policies that can be used directly in learning and planning but abstract away the details of low-level control.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “, 2017b) or options (Sutton et al., 1999; Bacon et al., 2017) from a single stream of experience.”
Value Prediction Network (2017, background): “Let xt be the observation or a history of observations for partially observable MDPs (henceforth referred to as just observation) and let ot be the option [33, 31, 25] at time t.”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “A variety of reinforcement learning architectures have focused on learning temporal abstractions, such as options (Sutton et al., 1999b), with policies that may maximise pseudo-rewards (Konidaris & Barreto, 2009; Silver & Ciosek, 2012).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “We use a continuous variant of the classic four rooms environment as a testbed (Sutton et al., 1999).”
Neural Approaches to Conversational AI (2018, positive): “ected agent, chooses primitive actions to complete the subtask. Such hierarchical decision making processes can be cast in the mathematical framework of options over Markov Decision Processes (MDPs) (Sutton et al., 1999b), where options generalize primitive actions to higher-level actions. This is a”
Neural Programmer-Interpreters (2015, background): “Related problems have been studied in the literature on hierarchical reinforcement learning ( e.g. , Dietterich (2000); Andre & Russell (2001); Sutton et al. (1999) and Schaul et al. (2015)), imitation and apprenticeship learning ( e.g. , Kolter et al. (2008) and Rothkopf & Ballard (2013)) and…”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, background): “Such hierarchical decision-making processes can be cast in the mathematical framework of options over Markov Decision Processes (MDPs) [Sutton et al., 1999], where options generalize primitive actions to higher-level actions.”
Deep learning in neural networks: An overview (2014, methodology): “While HRL frameworks such as Feudal RL (Dayan and Hinton, 1993) and options (Sutton et al., 1999b; Barto et al., 2004; Singh et al., 2005) do not directly address the problem of automatic subgoal discovery, HQ-learning (Wiering and Schmidhuber, 1998a) automatically decomposes POMDPs (Sec.”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Second, we show that by learning to control the Cartesian coordinates of the keypoints in the image plane we are able to learn skills or options [32] grounded in pixel observations, which is an important problem in reinforcement learning.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Temporal abstractions options framework [78]) may also be employed to simplify the process of selecting actions, where agents select options instead of low-level actions.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, methodology): “To combat this, hierarchical reinforcement learning was introduced to extend MDPs to semi-MDPs (Sutton et al., 1999), where options (or macro actions) are introduced on top of primitive actions to decompose the goal of a task into multiple subgoals.”
Graying the black box: Understanding DQNs (2016, methodology): “ters with a clear entrance and termination areas as options and interpret the agent policy there. For some options we are able to derive rules for initiation and termination (i.e., landmark options, (Sutton et al., 1999; Mann et al., 2015)). To summarize, the manual clustering methodology comprises ”