Stochastic Neural Networks for Hierarchical Reinforcement Learning

By Carlos Florensa, Yan Duan, Pieter Abbeel. 2016

378 citations recorded by the source corpus.

Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Reinforcement Learning with Deep Energy-Based Policies (2017, background): “The form of our sampler resembles the stochastic networks proposed in recent work on hierarchical learning (Florensa et al., 2017).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Recent work has investigated deep hierarchical agents [1, 48, 16, 31], building on methods that explicitly design a hierarchy over actions [32, 12], or utilize options [43, 35].”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, background): “Previous approaches have looked at how to enforce diversity in an agent’s behavior [12, 15, 9].”
Maximum a Posteriori Policy Optimisation (2018, methodology): “A number of other authors have considered the same perspective but in a model-free RL setting (Neumann, 2011; Peters et al., 2010a; Florensa et al., 2017; Daniel et al., 2016) or inverse RL problems (Ziebart et al., 2008).”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “[10] use hand designed state features and train the lower layers of the policy by maximising MI, and then tune the policy network’s upper layers for specific tasks.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, methodology): “Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016; Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “could be used for this stage, including off-the-shelf exploration bonuses [29, 4] or unsupervised reinforcement learning methods [9, 12].”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Florensa et al. (2017) trained a stochastic neural network with mutual information regularization to discover skills.”
Disentangling the independently controllable factors of variation by interacting with the world (2018, positive): “We use a bilinear operation bil ( a, b ) Florensa et al. [2017].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “This principle, proposed in prior work that utilized purely model-free unsupervised RL methods [14, 18, 17, 24, 71], aims to learn diverse skills via a discriminability objective: a good set of skills is one where it is easy to distinguish the skills from each other, which means they perform distinc”
Data-Efficient Hierarchical Reinforcement Learning (2018, methodology): “A number of works have suggested that semantically distinct behavior can be induced by training on a set of diverse tasks, and have suggested pre-training the lower-level policy on such tasks [18, 10], or training the multi-level hierarchical policy in a multi-task setup [11, 36].”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “…theoretic objectives on learning that incentivize increased option diversity and increased entropy in option selection [Gregor et al., 2016, Florensa et al., 2017, Hausman et al., 2018, Eysenbach et al., 2018, 1A review of this long and at times acrimonious debate is outside of the scope…”
Discrete Sequential Prediction of Continuous Actions for Deep RL (2017, methodology): “[7] Pieter Abbeel Carlos Florensa, Yan Duan.”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “Prior work has proposed to acquire discrete sets of lower-level skills through hand-speciﬁcation of objectives or bottle-necks (Florensa et al., 2017; Frans et al., 2017; Sutton et al., 1999) and top-down training of hierarchically-organized policies (Dayan & Hinton, 1992; Vezhnevets et al., 2017).”
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning (2017, background): “A few recent works in deep RL have used hierarchies to explore in sparse reward environments (Florensa et al., 2017; Heess et al., 2016).”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, background): “Related policy representations have been explored in prior work (Hausman et al., 2018; Florensa et al., 2017).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Recent HRL works have begun to tackle more difﬁcult control domains with both large state spaces and long planning horizons [24, 30, 59, 17, 36, 37].”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “, 2016), hand-designing state features and training the lower layer behaviors to maximize mutual information against these features (Florensa et al., 2017), or constructing diversityseeking priors that cause lower-layer primitives to take on different roles (Daniel et al.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “…policies, recent work in HRL has studied various facets of the problem, including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016; Tessler et al., 2017) and integrating…”
Stochastic Activation Actor Critic Methods (2019, methodology): “Similar SNNs have been employed in several hierarchical RL systems to embed complex skills in an abstract form for higher level tasks [10].”