Latent Space Policies for Hierarchical Reinforcement Learning

By Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, Sergey Levine. 2018

208 citations recorded by the source corpus.

We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acquires a range of diverse strategies via a maximum entropy reinforcement learning objective. Each layer is also augmented with latent random variables, which are sampled from a prior distribution during the training of that layer. The maximum entropy objective causes these latent variables to be incorporated into the layer's policy, and the higher level layer can directly control the behavior of the lower layer through this latent space. Furthermore, by constraining the mapping from latent variables to actions to be invertible, higher layers retain full expressivity: neither the higher layers nor the lower layers are constrained in their behavior. Our experimental evaluation demonstrates that we can improve on the performance of single-layer policies on standard benchmark tasks simply by adding additional layers, and that our method can solve more complex sparse-reward tasks by learning higher-level policies on top of high-entropy skills optimized for simple low-level objectives.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In another class of methods (Haarnoja et al., 2017, 2018a), the latent variable z is selected independently at each time step, and the policy π(a|s, z) has some simple unimodal form (e.g., a Gaussian distribution) conditioned on z, but a complex multimodal form when z is integrated out.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Another direction in this area is to learn a world model [4, 37, 36] that generates latent states [88, 38, 77].”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “This idea was introduced in earlier work in continual learning [46] (see also M.B. Ring, PhD thesis, University of Texas at Austin, 1994) and has shown promise with new research [47,48], but deserves further study at large scale.”
Soft Actor-Critic Algorithms and Applications (2018, background): “…where the optimal policy is independent of scaling of the reward function, in maximum entropy reinforcement learning the scaling factor has to be compensated by the choice a of suitable temperature, and a sub-optimal temperature can drastically degrade the performance (Haarnoja et al., 2018c).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “(122) Haarnoja et al. (2018) and Ward et al. (2019) use such a policy within the maximum entropy and soft actor-critic frameworks respectively.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, methodology): “…RL algorithms can effectively handle multi-modal action distributions; for example, this can be achieved for the Soft Actor Critic algorithm (Haarnoja et al., 2018b) using the reparametrization trick (Kingma & Ba, 2014) with a normalizing ﬂow (Rezende & Mohamed, 2015) as described in…”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “…policies through active interaction with an environment (Kupcsik et al., 2013; Heess et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Haarnoja et al., 2018a; Nachum et al., 2018; Chandak et al., 2019; Peng et al., 2019), and methods that learn temporally extended actions, also known…”
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018, methodology): “[6], a similar construction is used to train an agent end to end.”
Toward Robust Long Range Policy Transfer (2021, background): “Therefore, a common approach is getting the latent space by pre-training the policy on the pre-training task, and further transfer the policy to the downstream task (Haarnoja et al. 2018).”
Learning Transferable Knowledge Through Embedding Spaces (2019, background): “Recently, latent and embedding spaces have been found helpful in this area [266; 267], where similar to chapter 8 of this thesis, the goal is to enforce state-space of several RL tasks to share similar distributions in a latent space.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “…et al., 1999; Bacon et al., 2017; Vezh-nevets et al., 2017; Nachum et al., 2018; 2019; Xie et al., 2018), our approach resembles Heess et al. (2016); Hausman et al. (2018); Haarnoja et al. (2018a); Merel et al. (2019), in that a HL controller modulates a LL controller through a continuous channel.”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, unknown): “In related work Hausman et al. [2018], Haarnoja et al. [2018] learn hierarchical policies with continuous latent variables optimizing the entropy regularized objective.”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, unknown): “Similar to hierarachical models, a latent representation can first be learned using a set of pre-training tasks, before transferring to downstream tasks [14, 17].”
AMP (2021, background): “Once trained, this latent representation can be used to build a control hierarchy, where the latent space model acts as a low-level controller, and a separate high-level controller is trained to specify controls via the latent space [Florensa et al. 2017; Haarnoja et al. 2018; Lynch et al. 2020].”
Decoupled Greedy Learning of CNNs (2019, methodology): “Greedy objectives have recently been used in several applications in reinforcement learning (Haarnoja et al., 2018) and in ensemble methods like Boosting (Huang et al., 2018).”
Reinforcement Learning for Combinatorial Optimization: A Survey (2020, methodology): “The main idea of this work is to use a layered hierarchical RL framework inspired by [Haarnoja et al. , 2018], in which there are several layers, each representing policies of different complexity: from the lowest - making the solution satisfy the problem’s constraints, to the highest, which is…”