Executing your Commands via Motion Diffusion in Latent Space

By Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, Gang Yu. Computer Vision and Pattern Recognition, 2022

696 citations recorded by the source corpus.

We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors. Since human motions are highly diverse and have a property of quite different distribution from conditional modalities, such as textual descriptors in natural languages, it is hard to learn a probabilistic mapping from the desired conditional modality to the human motion sequences. Besides, the raw motion data from the motion capture system might be redundant in sequences and contain noises; directly modeling the joint distribution over the raw motion sequences and conditional modalities would need a heavy computational over-head and might result in artifacts introduced by the captured noises. To learn a better representation of the various human motion sequences, we first design a powerful Variational AutoEncoder (VAE) and arrive at a representative and low-dimensional latent code for a human motion sequence. Then, instead of using a diffusion model to establish the connections between the raw motion sequences and the conditional inputs, we perform a diffusion process on the motion latent space. Our proposed Motion Latent-based Diffusion model (MLD) could produce vivid motion sequences conforming to the given conditional inputs and substantially reduce the computational overhead in both the training and inference stages. Extensive experiments on various human motion generation tasks demonstrate that our MLD achieves significant improvements over the state-of-the-art methods among extensive human motion generation tasks, with two orders of magnitude faster than previous diffusion models on raw motion sequences.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/7694f004c67840d7f098b3612d4b3dabd915c116

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Qi Tian
3. Lingxi Xie
4. Wenyu Liu
5. Xiaopeng Zhang
6. Alan L. Yuille
7. Yi Yang
8. Luc Van Gool

In Control and Systems Engineering

1. Amit H. Bermano
2. Guy Tevet
3. Chuan Guo
4. Li Cheng
5. Sen Wang
6. Qi Tian
7. Yuxuan Mu
8. Yonatan Shafir

Citations in the corpus

• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “This user study involves 42 AMT users with master recognition, with the side-by-side comparisons between MoMask and each of the state-of-the-art methods including MDM [42], MLD [9] and T2M-GPT [49].”
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “In addition to generating 3D assets from text, there are methods where 3D diffusion models [1, 7, 10, 26, 55, 63, 72, 88, 89, 91] generate motion sequences based on text-motion data.”
• Human Motion Diffusion as a Generative Prior (2023, methodology): “More recently, MLD [Xin et al. 2022] learned a latent diffusion model, similar to LDM [Rom-bach et al. 2022b] , which enables generating motion latent code instead of the motion itself, and lets a larger and pre-trained motion generator translate it into the physical space.”
• MotionGPT: Human Motion as a Foreign Language (2023, neutral): “…is one of the most important motion generation tasks, due to the user-2 Methods Text-to-Motion Motion-to-Text Motion Prediction Motion In-between Random Motion Random Description T2M-GPT [57] MLD [52] TM2T [11] MDM [46] MotionDiffuse[58] MotionGPT (Ours) friendly and convenient language input.”