The Option Keyboard: Combining Skills in Reinforcement Learning
By André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan J. Hunt, Shibl Mourad, David Silver and 1 more. 2021
116 citations recorded by the source corpus.
The ability to combine known skills to create new ones may be crucial in the solution of complex reinforcement learning problems that unfold over extended periods. We argue that a robust way of combining skills is to define and manipulate them in the space of pseudo-rewards (or "cumulants"). Based on this premise, we propose a framework for combining skills using the formalism of options. We show that every deterministic option can be unambiguously represented as a cumulant defined in an extended domain. Building on this insight and on previous results on transfer learning, we show how to approximate options whose cumulants are linear combinations of the cumulants of known options. This means that, once we have learned options associated with a set of cumulants, we can instantaneously synthesise options induced by any linear combination of them, without any learning involved. We describe how this framework provides a hierarchical interface to the environment whose abstract actions correspond to combinations of basic skills. We demonstrate the practical benefits of our approach in a resource management problem and a navigation task involving a quadrupedal simulated robot.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Khimya Khetarpal
- Irina Rish
- Matthew Riemer
- Lucas Caccia
- Ignacio Cases
- Arslan Chaudhry
- Jorge Armando Mendez Mendez
- Liyuan Wang
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “(Sahni et al., 2017; Barreto et al., 2019) have focused on the natural combination of both skill and composition to allow for explicit reuse of previously acquired knowledge in the form of skills.”
- Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, unknown): “, 2018), and more recently, using pseudo-rewards (Barreto et al., 2019).”