Context-Specific Representation Abstraction for Deep Option Learning
By Marwa Abdulhai, Dong-Ki Kim, Matthew Riemer, Miao Liu, G. Tesauro, Jonathan P. How. 2021
16 citations recorded by the source corpus.
Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration. One promising approach that learns these options end-to-end is the option-critic (OC) framework. We examine and show in this paper that OC does not decompose a problem into simpler sub-problems, but instead increases the size of the search over policy space with each option considering the entire state space during learning. This issue can result in practical limitations of this method, including sample inefficient learning. To address this problem, we introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL), a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space. We test our method against hierarchical, non-hierarchical, and modular recurrent neural network baselines, demonstrating significant sample efficiency improvements in challenging partially observable environments.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Sergey Levine
- Joelle Pineau
- Eric Eaton
- Chelsea Finn
In Management Science and Operations Research
- Doina Precup
- Irina Rish
- Khimya Khetarpal
- Matthew Riemer
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- George Tucker
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Furthermore, in large environments with underlying Factored MDP structure, powerful abstractions have also been formed by leveraging context specific independencies that only hold for a subset of states (Abdulhai et al., 2022).”