Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

By Ofir Nachum, Shixiang Shane Gu, Honglak Lee, Sergey Levine. 2018

233 citations recorded by the source corpus.

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at this https URL).

Subfields: Artificial Intelligence, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “The framework can be extended to other types of environments, such as partially observable and high-dimensional ones, through, e.g., probabilistic or differentiable planning [49, 94, 80], latent embedding of (belief) states [36] and goals [67].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “While the connection between temporal contrastive learning and approximate dynamics models has appeared in previous works [30], it is not immediately clear how one should learn the approximate linear dynamics required by Theorem 3.”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Contrastive Objectives Beyond model-based representations, many previous works propose the use of contrastive losses as a way of learning useful state representations (Wu et al., 2018; Nachum et al., 2018; Srinivas et al., 2020; Stooke et al., 2020).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “Following the derivations in Achiam et al. (2017) and Nachum et al. (2018a), we express the performance of a policy π in M in terms of linear opterators: where R is a vector representation of the rewards of M , Π π is a linear operator mapping state distributions to state-action distributions…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, neutral): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering…”
Learning Latent Plans from Play (2019, methodology): “[32] derive a similar architecture to Play-LMP, justifying it in a hierarchical reinforcement learning setting.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Recent HRL works have begun to tackle more difﬁcult control domains with both large state spaces and long planning horizons [24, 30, 59, 17, 36, 37].”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, background): “Nachum et al. [22] propose a way to balance between representation efﬁciency and capabilities.”
Learning Domain Invariant Representations in Goal-conditioned Block MDPs (2021, background): “[47] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine.”
EMI: Exploration with Mutual Information (2018, background): “Nachum et al. (2018) connects mutual information estimators to represen- tation learning in hierarchical RL. Exploration with intrinsic motivation Prior works on exploration mostly employ intrinsic motivation to estimate the measure of novelty or surprisal to guide the exploration.”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, methodology): “Finally, we comment that the contrastive learning technique used in this paper has been used by other reinforcement learning algorithms for learning feature representation (e.g., Kim et al. (2019); Nachum et al. (2019); Srinivas et al. (2020)) without theoretical guarantee.”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “Hierarchical reinforcement learning combined with deep neural networks has received increased attention in the past several years [Osa et al., 2019, Nachum et al., 2018b, Kulkarni et al., 2016, Nachum et al., 2018a, Levy et al., 2018, Vezhnevets et al., 2017], mainly applied to efﬁcient training of…”
Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “…(Sutton, Precup, and Singh 1999; Bacon, Harb, and Precup 2017; Omidshafiei et al. 2018; Riemer, Liu, and Tesauro 2018; Riemer et al. 2020) and goal-conditioned hierarchies to learn abstract sub-goals (Nachum et al. 2019; Levy, Platt, and Saenko 2019; Kulkarni and Narasimhan 2016; Kim et al. 2020).”
Value Preserving State-Action Abstractions (2020, methodology): “(21) This theorem offers a clear path for extending the guarantees of φ-relative options beyond the typical two-timescale setup observed in recent work (Bacon et al., 2017; Nachum et al., 2019) to fully realize the benefits of (multi-level) hierarchical abstraction (Levy et al., 2019).”
Deep Hierarchical Planning from Pixels (2022, methodology): “…most HRL approaches require domain knowledge to break down tasks, either through manually speciﬁed subtasks (Tessler et al., 2017) or semantic goal spaces such as global XY coordinates for navigation tasks (Andrychowicz et al., 2017; Nachum et al., 2018a) or robot poses (Gehring et al., 2021).”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, background): “More recently, Nachum et al. (2019) provides some interesting guarantees in the context of hierarchical reinforcement learning, while Such et al. (2018) visualized the representations learned by Atari-playing agents.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Among various ways (Sutton et al., 1999; Bacon et al., 2017; Vezh-nevets et al., 2017; Nachum et al., 2018; 2019; Xie et al., 2018), our approach resembles Heess et al. (2016); Hausman et al. (2018); Haarnoja et al. (2018a); Merel et al. (2019), in that a HL controller modulates a LL controller…”
Hierarchical Policy Learning is Sensitive to Goal Space Design (2019, background): “…(Dayan & Hinton, 1993; Sutton et al., 1999) has been shown to be effective in dealing with long time scales, improving performance via temporal abstraction and learning skills transferable to new tasks (Vezhnevets et al., 2017; Frans et al., 2018; Florensa et al., 2017; Nachum et al., 2018b;a).”
Multitask Soft Option Learning (2019, methodology): “One widely used approach is to rely on additional human-designed input, often in the form of manually specified subgoals (Vezhnevets et al., 2017; Nachum et al., 2018) or a fixed temporal extension of learned skills (Frans et al., 2018).”
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, methodology): “In many HRL methods the high level learns to set subgoals for the low level [34, 35, 36, 15, 37, 14].”