DeepMimic

By Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne. 2018

588 citations recorded by the source corpus.

A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Implicit Behavioral Cloning (2021, background): “In addition to behavioral cloning (BC) [1], the machine learning and robotics communities have explored many additional approaches in imitation learning [51, 52, 53], often in ways that need additional information.”
Neural state machine for character-scene interactions (2019, methodology): “Peng et al. [2018a; 2017; 2018b]. cope with this problem by providing reference motion capture data [Peng et al. 2018a, 2017] or video data [Peng et al. 2018b].”
Backplay: "Man muss immer umkehren" (2018, unknown): “Many algorithms in this literature require access to the expert’s policy or use supervision from the expert’s actions and/or states (Nair et al., 2017; Hester et al., 2017; Ho & Ermon, 2016; Aytar et al., 2018; Lerer & Peysakhovich, 2018; Peng et al., 2018).”
Learning Navigation Subroutines from Egocentric Videos (2019, background): “Skills learned from such videos don’t depend on the perceptual input [4, 5].”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, background): “Work such as (Heess et al., 2017; Peng et al., 2018) considered agile but primarily short-horizon, reactive locomotion behavior.”
Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots (2021, methodology): “a) RL-based Control for Legged Robots: Reinforcement learning for legged locomotion has shown promising results in acquiring locomotion skills in simulation [18]– [20] and in the real world [21]–[23].”
Learning Montezuma's Revenge from a Single Demonstration (2018, result): “The amount of noise added is thus small, but comparable to the previous best results such as those by Pohlen et al. [2018] and Aytar et al.”
Dataset Distillation by Matching Training Trajectories (2022, background): “Imitation learning attempts to learn a good policy by observing a collection of expert demonstrations [29, 30, 31].”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, methodology): “We use reference state initialization (RSI) [29] during training and randomly select a starting point for a motion clip for imitation.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “transportation schedule [22, 53, 159, 253], traffic signal control [50, 59], energy management [180], wireless security [172], satellite docking [76], edge computing [299], chemical processes [241], video games [28, 153, 192, 211, 262, 300], board games of Go, shogi, chess and arcade game PAC-MAN [1”
Catch & Carry (2019, methodology): “2001], often also using motion capture as a reference for the controller [e.g. Yin et al. 2007; Liu et al. 2010; Peng et al. 2018].”
Deep reinforcement learning: a survey (2020, methodology): “Apart from Atari and simple agents in Mu-joco (e.g., half-cheetah, ant, and spider), DeepMimic (Peng et al., 2018a) further develops challenging mul-tiskilled agents including multiple characters (e.g., human, Atlas robot, bipedal dinosaur, and dragon) and a large variety of skills (e.g.,…”
Hierarchical Style-based Networks for Motion Synthesis (2020, background): “10, the right part is demonstration synthesized by our model (top) and the bottom one is learned policy with [26], where the learned motion succeeds in following the synthesized one.”
Goal-conditioned Imitation Learning (2019, background): “Furthermore, most tasks tackled with these methods consist on tracking expert state trajectories [30, 31], but cannot adapt to unseen situations.”
A Distributional View on Multi-Objective Policy Optimization (2020, methodology): “In order to balance these multiple objectives, prior work relied on heavily-tuned reward functions (e.g. Peng et al., 2018).”