Neural probabilistic motor primitives for humanoid control

By Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, Nicolas Manfred Otto Heess. 2018

187 citations recorded by the source corpus.

We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video ( this https URL ) summarizing our results.

Subfields: Cognitive Neuroscience, Social Psychology, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Social Psychology

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Critic Regularized Regression (2020, background): “[22] Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “However, there have been some recent variants of the above works (Lynch et al., 2020; Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) which extract skills from a prior dataset and using it for either performing tasks directly (Lynch et al., 2020) or learning downstream tasks…”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “Using fixedlength skills allows for scalable skill learning and has proven to be effective in prior works [22, 23, 9, 36, 21, 5].”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “Data description Note that for the purposes of data collection on the CMU humanoid tasks, we use expert policies trained according to Merel et al. (2019b), with only a single motor skill module from motion capture that is reused in each task.”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, background): “Previously, only an agent operating in the space of pre-learned motor primitives was able to solve the task from pixel observations (Merel et al., 2018; 2019); here we show that V-MPO can learn a challenging visuomotor task from scratch (Fig.”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, background): “Demonstrations can also be incorporated during pre-training to acquire more complex skills [26].”
From Motor Control to Team Play in Simulated Humanoid Football (2021, methodology): “Our agent brings together several of these ideas (33, 62, 63, 66) in a multi-scale learning architecture.”
Deep neuroethology of a virtual rodent (2019, methodology): “…deep reinforcement learning algorithms which now allow artificial agents to perform complex and adaptive movement in physical environments with sensory information that is increasingly similar to that available to animals (Peng et al., 2016; 2017; Heess et al., 2017; Merel et al., 2019a;c).”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “…et al., 1999; Bacon et al., 2017; Vezh-nevets et al., 2017; Nachum et al., 2018; 2019; Xie et al., 2018), our approach resembles Heess et al. (2016); Hausman et al. (2018); Haarnoja et al. (2018a); Merel et al. (2019), in that a HL controller modulates a LL controller through a continuous channel.”
Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives (2019, background): “Auxiliary tasks such as motion imitation have been applied to learn motor primitives that are capable of performing a variety of sophisticated skills [20, 28, 23, 22].”
Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real (2019, background): “Recent work uses efﬁcient policy cloning to learn generalized tracking policies via learned latent spaces [8].”
dm_control: Software and Tasks for Continuous Control (2020, methodology): “The Locomotion framework has enabled a number of research eﬀorts including Merel et al. (2017), Merel et al. (2019b), Merel et al. (2019a) and more recently has been employed to support Multi-Agent domains in Liu et al. (2019), Sunehag et al. (2019) and Banarse et al. (2019).”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, methodology): “These tasks are made up of the corridor locomotion tasks involving the CMU Humanoid [Tassa et al., 2020], for which prior efforts have either used motion capture data [Merel et al., 2019a,b] or training from scratch [Song et al., 2020].”
Catch & Carry (2019, methodology): “2016] as well as constrain the solutions that emerge [e.g. Peng et al. 2018; Merel et al. 2019c].”
AMP (2021, background): “…the latent representation can be trained to encode behaviors from reference motion clips, which then constrains the behavior of a character to be similar to those observed in the motion data, therefore leading to more natural behaviors for downstream tasks [Merel et al. 2019; Peng et al. 2019a].”
Toward Robust Long Range Policy Transfer (2021, methodology): “To encourage the latent space is diverse enough, one solution is to leverage reference data 4 to pre-train the latent space (Merel et al. 2019).”