Learning human behaviors from motion capture by adversarial imitation

By Josh Merel, Yuval Tassa, TB Dhruva, Sriram Srinivasan, Jay Lemmon, Ziyun Wang, Greg Wayne, Nicolas Manfred Otto Heess. 2017

221 citations recorded by the source corpus.

Rapid progress in deep reinforcement learning has made it increasingly feasible to train controllers for high-dimensional humanoid bodies. However, methods that use pure reinforcement learning with simple reward functions tend to produce non-humanlike and overly stereotyped movement behaviors. In this work, we extend generative adversarial imitation learning to enable training of generic neural network policies to produce humanlike movement patterns from limited demonstrations consisting only of partially observed state features, without access to actions, even when the demonstrations come from a body with different and unknown physical parameters. We leverage this approach to build sub-skill policies from motion capture data and show that they can be reused to solve tasks when controlled by a higher level controller.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Imitating Interactive Intelligence (2020, background): “Further, it is at present intrinsically difficult to do so because algorithms and systems have not yet been developed that can perceive and understand the intentions of humans in a way that transfers across radical changes in embodiment, environment, and perspective (Stadie et al., 2017; Borsa et al”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “A method of this type uses GANs [Goodfellow et al. , 2014] to bring the state distribution of the imitator closer to that of the trainer [Merel et al. , 2017].”
DeepMind Control Suite (2018, methodology): “Humanoid_CMU (124, 56, 137): A humanoid body with 56 joints, adapted from (Merel et al., 2017) and based on the ASF model of subject #8 in the CMU Motion Capture Database.”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “Inspired by reward augmentation as described in Li et al. [25] and Merel et al. [28], we provide additional guidance via a hybrid reward function that combines the imitation reward r gail with the task reward r task : Maximizing this hybrid reward can be interpreted as simultaneous reinforcement…”
Neural probabilistic motor primitives for humanoid control (2018, methodology): “As proposed in Merel et al. (2017); Peng et al. (2018), episodes are initialized to poses throughout the motion capture reference and episodes are early-terminated when the character falls.”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, neutral): “Veˇcerík et al. (2017), Merel et al. (2017) and Paine et al. (2018) have demonstrated similar results in robotics.”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, methodology): “Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) were orignally used for image generation but have now been successfully applied to model audio, text and motor behaviors (Ho & Ermon, 2016; Merel et al., 2017).”
Task-Relevant Adversarial Imitation Learning (2019, background): “Although many variants are introduced in the literature [Li et al., 2017, Fu et al., 2018, Merel et al., 2017, Zhu et al., 2018, Baram et al., 2017], making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.”
dm_control: Software and Tasks for Continuous Control (2020, methodology): “The Locomotion framework has enabled a number of research eﬀorts including Merel et al. (2017), Merel et al. (2019b), Merel et al. (2019a) and more recently has been employed to support Multi-Agent domains in Liu et al. (2019), Sunehag et al. (2019) and Banarse et al. (2019).”
Composing Complex Skills by Learning Transition Policies (2018, background): “Learning continuous control of diverse behaviors in locomotion (Merel et al., 2017; Heess et al., 2017; Peng et al., 2017) and robotic manipulation (Ghosh et al., 2018) is an active research area in reinforcement learning (RL).”
Generative Adversarial Imitation from Observation (2018, background): “There are also other recent techniques that address imitation from observation [20, 34], but these approaches have thus far focused mostly on experimentation and less on the theoretical underpinnings of the control problem.”
Catch & Carry (2019, methodology): “Task-driven learning can also occur in combination with motion capture which constrains behavior to be more humanlike [Peng et al. 2017, 2018; Merel et al. 2017].”
Hierarchical visuomotor control of humanoids (2018, methodology): “We use a 56 degree-of-freedom (DoF) humanoid body that was developed in previous work (Merel et al., 2017), a version of which is available with motion-capture playback in the DeepMind control suite (Tassa et al., 2018).”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, methodology): “Approaches based on GAIL [14] have also been proposed to eliminate the need for manual reward engineering [33, 60, 67].”
AMP (2021, methodology): “AMP is able to closely imitate a large variety of highly dynamic skills, while also avoiding many of the visual artifacts exhibited by prior adversarial motion imitation systems [Merel et al. 2017; Wang et al. 2017].”
Imitation Learning from Video by Leveraging Proprioception (2019, methodology): “Another approach of this type is the work of Merel et al. [2017], which is concerned instead with single state distributions.”