Data-Efficient Hierarchical Reinforcement Learning

By Ofir Nachum, Shixiang Shane Gu, Honglak Lee, Sergey Levine. 2018

1,023 citations recorded by the source corpus.

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher- and lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We find that our resulting HRL agent is generally applicable and highly sample-efficient. Our experiments show that our method can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, methodology): “The policy-learning phase in our framework uses the paradigm of goal-conditioned HRL [57, 21, 68, 90].”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Recent work has investigated deep hierarchical agents [1, 48, 16, 31], building on methods that explicitly design a hierarchy over actions [32, 12], or utilize options [43, 35].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “A particularly promising approach was proposed in [15] and [16], using goal conditioned policies at multiple layers of hierarchy for RL.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “Following the derivations in Achiam et al. (2017) and Nachum et al. (2018a), we express the performance of a policy π in M in terms of linear opterators: where R is a vector representation of the rewards of M , Π π is a linear operator mapping state distributions to state-action distributions…”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “The high-level goal selection mechanism chooses goal states for the low-level to try and reach (similar to [27]).”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, neutral): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering…”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “We plan to extend the algorithm to work with off-policy data, potentially using relabelling tricks (Andrychowicz et al., 2017; Nachum et al., 2018) and explore more nuanced planning algorithms.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…to sample efﬁciency or data efﬁciency (Precup et al., 2001; Degris et al., 2012; Gu et al., 2017b; Haarnoja et al., 2018; Lillicrap et al., 2016; Nachum et al., 2018), which measures the amount of environment interactions incurred during training, without regard to how many distinct policies…”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, methodology): “, 2017), or not learned them at all (Levy et al., 2017; Nachum et al., 2018), we take a principled approach to representation learning in hierarchical RL, translating a bound on sub-optimality to a practical learning objective.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “One class of methods (Bacon et al., 2017; Frans et al., 2017; Kaelbling, 1993a; Kulkarni et al., 2016; Nachum et al., 2018; Parr and Russell, 1998; Precup, 2000; Sutton et al., 1999; Vezhnevets et al., 2017) jointly learn a low-level policy for performing each of the skills together with a high-leve”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “, sub-goal) via hierarchical reinforcement learning (HRL) [26].”
Flexible Option Learning (2021, background): “This can be done in various ways such as identifying bottleneck states [McGovern and Barto, 2001, Şimşek et al., 2005, Menache et al., 2002, Machado et al., 2017] or through feudal RL [Dayan and Hinton, 1993, Vezhnevets et al., 2017, Nachum et al., 2018].”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, methodology): “Existing use cases suggest reinforcement learning [9], solving a Jigsaw puzzle or image colorization [4, 19].”
CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning (2018, background): “Yet, while hierarchical RL agents choose their own subgoals, they usually do so to achieve higher-level goals/tasks imposed by the engineer (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2018).”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, methodology): “The subgoals can be set directly in the same space as the original goal [15], or in an embedding space [21].”
Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, methodology): “We use this as a lens to study how the different algorithms handle HRL in this setting.”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, methodology): “But to keep with the hierarchical learning perspective, standard HRL methods have been extended to the deep learning context with various frameworks [113, 114, 115, 116, 117].”
Prospection: Interpretable plans from language by predicting the future (2019, background): “This has proven particularly important for RL-based approaches [13] and interpretability [14].”