When Waiting is not an Option : Learning Options with a Deliberation Cost

By Jean Harb, Pierre-Luc Bacon, Martin Klissarov, Doina Precup. 2017

170 citations recorded by the source corpus.

Recent work has shown that temporally extended actions (options) can be learned fully end-to-end as opposed to being specified in advance. While the problem of how to learn options is increasingly well understood, the question of what good options should be has remained elusive. We formulate our answer to what good options should be in the bounded rationality framework (Simon, 1957) through the notion of deliberation cost. We then derive practical gradient-based learning algorithms to implement this objective. Our results in the Arcade Learning Environment (ALE) show increased performance and interpretability.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “While much of the work on learning options has focused on single task learning (Bacon et al., 2017; Machado et al., 2017a,b; Harb et al., 2018; Khetarpal et al., 2020a; Riemer et al., 2020; Klissarov and Precup, 2021), work on discovering options in a multi-task context has shown some promising pote”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “In practice, regularizers are essential to learn multiple effective and temporally abstracted sub-policies [2, 16, 42].”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Hard-coded abstractions often lack modeling ﬂexibility and are task-speciﬁc [57, 29, 24, 41], while learned abstractions often ﬁnd degenerate solutions without careful tuning [4, 22].”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, background): “A number of works have proposed practical and scalable algorithms for learning option policies with reinforcement learning [e.g. Bacon et al., 2017, Zhang and Whiteson, 2019, Smith et al., 2018] or criteria for option induction [e.g. Harb et al., 2018, Harutyunyan et al., 2019].”
Context-Aware Policy Reuse (2018, result): “In contrast to the works of option discovery [4, 16, 17, 19], CAPS focuses on multi-policy reuse.”
Composable Planning with Attributes (2018, methodology): “In our experiments below, we compare our method to Option-Critic with a deliberation cost (Harb et al., 2017), a hierarchical reinforcement learning method that learns options without additional supervision by including a reward when a target goal is achieved.”
Multitask Soft Option Learning (2019, methodology): “It can also be seen as a form of deliberation cost (Harb et al., 2017) as it is only nonzero whenever we terminate an option and the master policy needs to select another to execute: if the option is not terminated, we have zt = zt 1 with probability 1 for both prior and posterior by construction…”
Learning Abstract Options (2018, methodology): “We follow [6] to extend A3C to the Asynchronous Advantage Option-Critic (A2OC) and Asynchronous Advantage Hierarchical Option-Critic architectures (A2HOC).”
Flexible Option Learning (2021, methodology): “This framework has lead to recent approaches [Levy and Shimkin, 2011, Bacon et al., 2016, Harb et al., 2017, Riemer et al., 2018, Tiwari and Thomas, 2018] that rely on policy gradient methods to learn the option policies.”
Discovering Options for Exploration by Minimizing Cover Time (2019, background): “, 2002; Konidaris & Barto, 2009), while other approaches use gradient descent to generate options using the observed rewards (Mankowitz et al., 2016; Bacon et al., 2017; Harb et al., 2017).”
Options of Interest: Temporal Abstraction with Interest Functions (2020, background): “This aspect of options has been ignored since, including in recent works (Bacon, Harb, and Precup 2017; Harb et al. 2017; Harutyunyan et al. 2019a; 2019b) because there was no elegant way to learn initiation sets.”