Character controllers using motion VAEs

By Hung Yu Ling, Fabio Zinno, George Cheng, Michiel van de Panne. ACM Transactions on Graphics, 2020

320 citations recorded by the source corpus.

A fundamental problem in computer animation is that of realizing purposeful and realistic human movement given a sufficiently-rich set of motion capture clips. We learn data-driven generative models of human movement using autoregressive conditional variational autoencoders, or Motion VAEs. The latent variables of the learned autoencoder define the action space for the movement and thereby govern its evolution over time. Planning or control algorithms can then use this action space to generate desired motions. In particular, we use deep reinforcement learning to learn controllers that achieve goal-directed movements. We demonstrate the effectiveness of the approach on multiple tasks. We further evaluate system-design choices and describe the current limitations of Motion VAEs.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/c3adba29b9ff94a97d56a21003df333ccfc25e08

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Executing your Commands via Motion Diffusion in Latent Space (2022, background): “For runtime, training tasks 8 hours for VAEs V and 4 hours for denoiser (cid:15) θ on 8 Tesla V100 GPUs, and we test MLD with a single V100 in Sec.”
AMP (2021, methodology): “Data-driven methods using generative models, such as Gaussian processes [Levine et al. 2012; Ye and Liu 2010] and neural networks [Holden et al. 2017; Ling et al. 2020; Zhang et al. 2018], have also been applied to synthesize motions online.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, methodology): “VPoser-t fails to produce any plausible lower-body motion since it uses only a pose prior, while using MVAE as the motion prior often gives unnatural and implausible motions that don’t align well with the observed keypoints.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “There exists work to capture humans in a static 3D scene [33], even without using external cameras [30], and work to synthesize static poses [34, 49], or full body movement [32, 32, 50, 68] in a 3D scene.”
EDGE: Editable Dance Generation From Music (2022, background): “These approaches are often capable of generating diverse motions, but often fall short in capturing the physical laws governing human movement or rely on difficult-to-train reinforcement learning solutions [34, 60].”