Learning Abstract Options

By Matthew Riemer, Miao Liu, G. Tesauro. 2018

85 citations recorded by the source corpus.

Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework (Sutton et al., 1999). However, only recently in (Bacon et al., 2017) was a policy gradient theorem derived for online learning of general purpose options in an end to end fashion. In this work, we extend previous work on this topic that only focuses on learning a two-level hierarchy including options and primitive actions to enable learning simultaneously at multiple resolutions in time. We achieve this by considering an arbitrarily deep hierarchy of options where high level temporally extended options are composed of lower level options with finer resolutions in time. We extend results from (Bacon et al., 2017) and derive policy gradient theorems for a deep hierarchy of options. Our proposed hierarchical option-critic architecture is capable of learning internal policies, termination conditions, and hierarchical compositions over options without the need for any intrinsic rewards or subgoals. Our empirical results in both discrete and continuous environments demonstrate the efficiency of our framework.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…work on discovering options in a multi-task context has shown some promising potential both theoretically (Brunskill and Li, 2014) and empirically (Achiam, Edwards, Amodei, and Abbeel, 2018; Riemer, Liu, and Tesauro, 2018; Igl, Gambardella, He, Nardelli, Siddharth, Böhmer, and Whiteson, 2019).”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “, 2017) or deep (Riemer et al., 2018) hierarchies of options in essence learn very complex patterns of weight sharing over time.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “In particular, challenges of routing are related to those experienced by end to end architectures for learning sequencing between policies over time with neural networks [Bacon et al., 2017] and extensions to hierarchical policies [Riemer et al., 2018].”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “HRL is a structured framework with multi-level reasoning and extended temporal abstraction [2, 8, 9, 17, 28, 29, 33, 38].”
On the Role of Weight Sharing During Deep Option Learning (2019, methodology): “…Policy Gradient Theorem Remarks About the Augmented Process Directly following the original hierarchical option-critic policy gradient theorem proof (Riemer et al., 2018), we begin by making a remark about augmented processes which will be helpful later in the ﬁnal steps of the…”
Flexible Option Learning (2021, methodology): “This framework has lead to recent approaches [Levy and Shimkin, 2011, Bacon et al., 2016, Harb et al., 2017, Riemer et al., 2018, Tiwari and Thomas, 2018] that rely on policy gradient methods to learn the option policies.”
CompILE: Compositional Imitation Learning and Execution (2018, background): “, 2018) and the options framework (Sutton et al., 1999; Kulkarni et al., 2016; Bacon et al., 2017; Riemer et al., 2018) similarly deal with learning segmentations and representations of behavior, but in a purely generative way.”
Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “Notably, the option-critic (OC) framework has become popular because it can learn deep hierarchies of both option policies and termination conditions from scratch without domain-specific knowledge (Bacon, Harb, and Precup 2017; Riemer, Liu, and Tesauro 2018).”