Options of Interest: Temporal Abstraction with Interest Functions
By Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, Doina Precup. 2020
52 citations recorded by the source corpus.
Temporal abstraction refers to the ability of an agent to use behaviours of controllers which act for a limited, variable amount of time. The options framework describes such behaviours as consisting of a subset of states in which they can initiate, an internal policy and a stochastic termination condition. However, much of the subsequent work on option discovery has ignored the initiation set, because of difficulty in learning it from data. We provide a generalization of initiation sets suitable for general function approximation, by defining an interest function associated with an option. We derive a gradient-based learning algorithm for interest functions, leading to a new interest-option-critic architecture. We investigate how interest functions can be leveraged to learn interpretable and reusable temporal abstractions. We demonstrate the efficacy of the proposed approach through quantitative and qualitative results, in both discrete and continuous environments.
Subfields: Control and Systems Engineering, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Control and Systems Engineering
- Pieter Abbeel
- Sergey Levine
- Doina Precup
- Igor Mordatch
- Jonathan P. How
- G. Tesauro
- Miao Liu
- Aditya Grover
In Management Science and Operations Research
- Pieter Abbeel
- Doina Precup
- Igor Mordatch
- Sergey Levine
- Kevin Lu
- Matthew Riemer
- David Abel
- G. Tesauro
Citations in the corpus
- Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “Our work is most similar to Khetarpal et al. (2020) which learns a smaller initiation set through an attention mechanism to better focus an option to a region of the state space, and hence achieve specialization of skills.”
- Flexible Option Learning (2021, methodology): “We also compare our approach, MOC, to the interest option-critic (IOC) [Khetarpal et al., 2020] which has been shown to outperform the option-critic algorithm derived for the continuous control setting [Harb et al., 2017, Klissarov et al., 2017].”
- Reset-Free Lifelong Learning with Skill-Space Planning (2020, methodology): “Our skill-practice distribution somewhat resembles the interest function of Khetarpal et al. (2020); both are designed to associate certain skills with certain states, forming a type of curriculum.”
- What can I do here? A Theory of Affordances in Reinforcement Learning (2020, background): “, 2019), where intents are akin to subgoals and affordances could then take the role of initiation sets for options (Khetarpal et al., 2020).”