Learning Multi-Level Hierarchies with Hindsight

By Andrew Levy, George Dimitri Konidaris, Robert W. Platt, Kate Saenko. 2017

382 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, the concept of hindsight has been extended to both hierarchical (Levy et al., 2017) and modular (Colas et al.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering…”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, background): “…goal-conditioned hierarchical designs have either learned representations naively end-to-end (Vezhnevets et al., 2017), or not learned them at all (Levy et al., 2017; Nachum et al., 2018), we take a principled approach to representation learning in hierarchical RL, translating a bound on…”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, methodology): “Similar to previous HRL approaches [15, 18], one component of the high-level action, a HL t , is the subgoal g LL t ∈ G LL . g LL t indicates a desired relative change of certain mutable components of the observation, denoted as x t (cid:48) , yielding a subgoal x t (cid:48) + g LL t for the…”
Generalized Hindsight for Reinforcement Learning (2020, methodology): “Several hierarchical methods (Levy et al., 2017; Nachum et al., 2018) train a low-level policy to achieve subgoals and a higherlevel controller to propose those subgoals.”
Value Preserving State-Action Abstractions (2020, background): “…levels of granularity (Dayan and Hinton, 1993; Wiering and Schmidhuber, 1997; Parr and Russell, 1998; Dietterich, 2000a; Barto and Mahadevan, 2003; Jong and Value Preserving State-Action Abstractions Stone, 2008; Konidaris, 2016; Bai and Russell, 2017; Konidaris et al., 2018; Levy et al., 2019).”
Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? (2019, methodology): “…options framework (Precup, 2000) and goal-conditioned hierarchies (Nachum et al., 2018b), due to their impressive success in recent work (Frans et al., 2018; Levy et al., 2017; Nachum et al., 2018a; 2019), though an examination of other architectures is an important direction for future research.”
Learning to Learn: Hierarchical Meta-Critic Networks (2019, background): “[14] present a novel approach to hierarchical reinforcement learning called Hierarchical Actor-Critic (HAC) that enables agents to learn to break down problems involving continuous action spaces into simpler sub-problems belonging to different time scales.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “This baseline is representative of a large class of prior work that uses hindsight relabeling [6, 76, 102, 106].”
Deep Hierarchical Planning from Pixels (2022, methodology): “HAC (Levy et al., 2017) learned interpretable hierarchies but required semantic goal spaces.”
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, methodology): “Previous attempts to alleviate this include off-policy corrections [15], hindsight subgoal sampling [14] and low-level policy pre-training [34].”
ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, methodology): “Previous attempts to alleviate this include off-policy corrections [15], hindsight subgoal sampling [14] and low-level policy pre-training [35].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, unknown): “This approach naturally fits recent research using deep neural networks for policy learning [272, 404, 241], which are sometimes structured to be successively executable in a manner similar to skill chaining [228], but have been used in other more compact representations [77, 78].”