Meta Learning Shared Hierarchies

By Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman. 2017

382 citations recorded by the source corpus.

We develop a metalearning approach for learning hierarchically structured policies, improving sample efficiency on unseen tasks through the use of shared primitives---policies that are executed for large numbers of timesteps. Specifically, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies. We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks. We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies. We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes. We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

Subfields: Artificial Intelligence, Biomedical Engineering, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Software

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016; Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).”
Modular meta-learning (2018, methodology): “Frans et al. [19] jointly optimize a master policy and a set of sub-policies (options) that can be used to solve new problems; this method can be seen as having a single ﬁxed scheme of combination via the master policy; it is in contrast to our ability to handle a variety of computational…”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Much recent met-alearning work [6, 26, 27, 31, 37, 38, 53, 56, 60, 64, 79] focuses on domains where achieving zero-shot generalization is unclear and follow a common paradigm which assumes that the shared representation of monolithic architectures (with the exception of concurrent work [2]) can be…”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Another difference from most HRL work [10, 11] is that we use off-policy learning, leading to signiﬁcant improvements in sample efﬁciency.”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, neutral): “Prior work has proposed to acquire discrete sets of lower-level skills through hand-speciﬁcation of objectives or bottle-necks (Florensa et al., 2017; Frans et al., 2017; Sutton et al., 1999) and top-down training of hierarchically-organized policies (Dayan & Hinton, 1992; Vezhnevets et al., 2017).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “The ﬁrst aims to learn effective low-level policies end-to-end directly from ﬁnal task rewards with minimal human engineering, such as through the option-critic architecture [4, 22] or multi-task or meta learning [18, 52].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “One class of methods (Bacon et al., 2017; Frans et al., 2017; Kaelbling, 1993a; Kulkarni et al., 2016; Nachum et al., 2018; Parr and Russell, 1998; Precup, 2000; Sutton et al., 1999; Vezhnevets et al., 2017) jointly learn a low-level policy for performing each of the skills together with a…”
Multitask Soft Option Learning (2019, methodology): “MLSH (Frans et al., 2018) is a multitask option learning algorithm like MSOL, but utilizes ‘hard’ options for both learning and transfer, i.”
TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game (2018, background): “Although many papers have been devoted to automatically learning meaningful hierarchies in a Markovian Decision Process [33, 34, 35, 36, 31, 37], none of these methods can work eﬃciently on environments as complex as SC2.”
Deep learning for cardiovascular medicine: a practical primer. (2019, background): “Capsule-based neural network, 32,33 meta-learning (learning how the network learns), 34–36 DeepMath (learn mathematical proofs), 37 or self-play (two agents learn by win and loss) 34 S2 ).”
Toward Robust Long Range Policy Transfer (2021, methodology): “Some works propose a hierarchical policy (Frans et al. 2018; Bacon, Harb, and Precup 2017; Peng et al. 2019), which contains a combination function to control how to select or combine a set of primitives.”
Learning Multi-Level Hierarchies with Hindsight (2017, background): “ subgoal layers may need to learn their sequences before the lower-level layers can fully learn theirs. 9 5 Related Work Hierarchical RL is a topic of ongoing research [10], [3], [12], [4], [2], [8], [5]. Much of the literature builds off the options framework [10], which generally uses a hierarchy ”
ProMP: Proximal Meta-Policy Search (2018, neutral): “, 2018) and meta-learning modular policies (Frans et al., 2018; Alet et al., 2018).”
Continual and Multi-task Reinforcement Learning With Shared Episodic Memory (2019, methodology): “The second is to use meta-learning setting (Duan et al., 2016; Wang et al., 2016; Frans et al., 2017) to optimize the agent over a series of tasks in the same environment.”
Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? (2019, methodology): “…– the options framework (Precup, 2000) and goal-conditioned hierarchies (Nachum et al., 2018b), due to their impressive success in recent work (Frans et al., 2018; Levy et al., 2017; Nachum et al., 2018a; 2019), though an examination of other architectures is an important direction for…”