Hierarchical Policy Learning is Sensitive to Goal Space Design

By Zach Dwiel, Madhavun Candadai, Mariano Phielipp, Arjun K. Bansal. 2019

15 citations recorded by the source corpus.

Hierarchy in reinforcement learning agents allows for control at multiple time scales yielding improved sample efficiency, the ability to deal with long time horizons and transferability of sub-policies to tasks outside the training distribution. It is often implemented as a master policy providing goals to a sub-policy. Ideally, we would like the goal-spaces to be learned, however, properties of optimal goal spaces still remain unknown and consequently there is no method yet to learn optimal goal spaces. Motivated by this, we systematically analyze how various modifications to the ground-truth goal-space affect learning in hierarchical models with the aim of identifying important properties of optimal goal spaces. Our results show that, while rotation of ground-truth goal spaces and noise had no effect, having additional unnecessary factors significantly impaired learning in hierarchical models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “the need of a well-calibrated latent goal space (Higgins et al., 2017; Dwiel et al., 2019; Nachum et al., 2018a); Section 6 briefly speculates on extensions of our framework to other environments as future directions.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “…decomposition of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012;…”