Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

By Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Shane Gu, Honglak Lee, Sergey Levine. 2019

122 citations recorded by the source corpus.

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

Subfields: Artificial Intelligence, Management Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

Citations in the corpus

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “…(Silver et al., 2016; Tesauro, 1995; Brown & Sandholm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can act in the environment and sample as many transitions and rewards as needed.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “…including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016; Tessler et al., 2017) and integrating goal-conditioned behaviors (Ghosh et al., 2019; Nachum et al., 2019).”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “…an unsupervised reward and composed via MPC, which has the promise of more explicit long-term hierarchical beneﬁts, whereas there is some evidence (Nachum et al., 2019) that policy-based hierarchy only aids in exploration – not long-term reasoning – and is replaceable with monolithic exploration…”
ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, background): “While the exploration challenge may be mitigated by a hierarchical structure [14, 15, 16, 17], the RL agent still dedicates a large portion of its training steps to learning to move towards spatial points of interest without collisions from scratch.”
Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation (2020, result): “Although this simple paradigm has achieved impressive results on hard exploration environments [25], in our initial experiments for off-policy evaluation we found the naive ensembling approach to yield poor confidence intervals.”
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, background): “Hierarchical Reinforcement Learning: In HRL solutions, usually a high-level policy uses a low-level one to generate primitive actions for extended time periods in order to facilitate better exploration [16].”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “Hierarchical reinforcement learning [4, 5, 18] works to temporally extend planning at lower levels, reducing the horizon the high-level policy must reason over [19].”