Learning World Graphs to Accelerate Hierarchical Reinforcement Learning
By Wenling Shang, Alexander R. Trott, Stephan Zheng, Caiming Xiong, Richard Socher. 2019
23 citations recorded by the source corpus.
In many real-world scenarios, an autonomous agent often encounters various tasks within a single complex environment. We propose to build a graph abstraction over the environment structure to accelerate the learning of these tasks. Here, nodes are important points of interest (pivotal states) and edges represent feasible traversals between them. Our approach has two stages. First, we jointly train a latent pivotal state model and a curiosity-driven goal-conditioned policy in a task-agnostic manner. Second, provided with the information from the world graph, a high-level Manager quickly finds solution to new tasks and expresses subgoals in reference to pivotal states to a low-level Worker. The Worker can then also leverage the graph to easily traverse to the pivotal states of interest, even across long distance, and explore non-locally. We perform a thorough ablation study to evaluate our approach on a suite of challenging maze tasks, demonstrating significant advantages from the proposed framework over baselines that lack world graph knowledge in terms of performance and efficiency.
Subfields: Artificial Intelligence, Reinforcement Learning, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Anders Jonsson
- Lorenzo Steccanella
- Kostas Daniilidis
- Jingyun Yang
- Shenghao Zhou
- Damien Allonsius
- Andrew Jaegle
- Konstantinos G. Derpanis
In Reinforcement Learning
- Joseph J. Lim
- Anders Jonsson
- Lorenzo Steccanella
- Kostas Daniilidis
- Jingyun Yang
- Shenghao Zhou
- Damien Allonsius
- Andrew Jaegle
In Graph Learning
Citations in the corpus
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Other examples of hybrid symbolic and sub-symbolic methods where a knowledge-base tool or graph-perspective enhances the neural (e.g., language [308] ) model are in [309,310] .”
- Hierarchical reinforcement learning for efficient exploration and transfer (2020, methodology): “Shang et al. (2019) use variational inference to construct a world graph similar to our region space.”
- Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “Concurrently to our work, Shang et al. (2019) propose a keyframing method that learns to select frames that are informative about the action trajectory.”