Learning and Transfer of Modulated Locomotor Controllers

By Nicolas Manfred Otto Heess, Greg Wayne, Yuval Tassa, Timothy P. Lillicrap, Martin A. Riedmiller, David Silver. 2016

223 citations recorded by the source corpus.

We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at this https URL

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Emergence of Locomotion Behaviours in Rich Environments (2017, methodology): “Observations The agents receive two sets of observations [22]: (1) a set of egocentric, “proprioceptive” features containing joint angles and angular velocities; for the Quadruped and Humanoid these features also contain the readings of a velocimeter, accelerometer, and a gyroscope positioned at the”
Reinforcement Learning with Deep Energy-Based Policies (2017, neutral): “This pretraining is similar in some ways to recent work on modulated controllers (Heess et al., 2016) and hierarchical models (Florensa et al., 2017).”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “LSTM was already used in reinforcement learning [112] for advantage learning [4], for constructing a potential function for reward shaping by representing the return by a sum of LSTM outputs across an episode [124], and learning policies [44, 80, 45].”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016; Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Heess et al. (2016) formulated the actions of the meta controller as continuous variables that are used to modulate the behavior of the low-level controller.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, neutral): “However, in contrast to end-to-end HRL approaches [30, 54], we can leverage a stable, two-phase learning setup.”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “To guarantee learning useful sub-policies, recent work has studied approaches that provide auxiliary rewards for the low-level policies [5, 18, 22, 40, 10].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “Recently, Heess et al. (2016) have independently proposed to learn a range of skills in a pre-training environment that will be useful for the down-stream tasks, which is similar to our framework.”
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning (2017, background): “A few recent works in deep RL have used hierarchies to explore in sparse reward environments (Florensa et al., 2017; Heess et al., 2016).”
Emergent Coordination Through Competition (2019, background): “, 2012) which embeds soccer in a wider universe of possible environments with consistent simulated physics, already used extensively in the machine learning research community (Heess et al., 2016; 2017; Bansal et al., 2017; Brockman et al., 2016; Tassa et al., 2018; Riedmiller et al., 2018).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Recent HRL works have begun to tackle more difﬁcult control domains with both large state spaces and long planning horizons [24, 30, 59, 17, 36, 37].”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, methodology): “This provides a small amount of domain knowledge, but this type of domain knowledge is much easier to specify than true intermediate goals or modulation (Heess et al., 2016; Kulkarni et al., 2016), and the entropy maximization term in the objective automatically causes the low-level policy to learn ”
Multi-Level Discovery of Deep Options (2017, methodology): “L G ] 5 O c t 2017 generalize well to multi-level hierarchies [2, 3, 4], while others are inefﬁcient for learning expressive representations [5, 6, 7, 8].”
Learning to Solve a Rubik’s Cube with a Dexterous Hand (2019, background): “In standard hierarchical reinforcement learning [11], [12], [13], [14] one higher-level policy and one lower-level policy policy are trained.”
Multitask Soft Option Learning (2019, background): “Several previous works try to overcome the restrictive nature of options that can lead to sub-optimal solutions by allowing the higher-level actions to modulate the behavior of the lower-level policies Heess et al. (2016); Haarnoja et al. (2018).”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “It is important to note that module composition takes two different forms with Policy Sketches and the architectures of (Heess et al., 2016) and (Devin et al., 2017): subpolicies in Policy Sketches behave as in hierarchical reinforcement learning (Kulkarni et al., 2016), where a subpolicy is chosen…”
Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement (2018, background): “Among these, a recurrent theme is the existence of sub-networks specialised in different skills that are managed by another network (Heess et al., 2016; Frans et al., 2017; Oh et al., 2017).”