Generating Diverse and Natural 3D Human Motions from Text

By Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, Li Cheng. Computer Vision and Pattern Recognition, 2022

1,034 citations recorded by the source corpus.

Automated generation of 3D human motions from text is a challenging problem. The generated motions are expected to be sufficiently diverse to explore the text-grounded motion space, and more importantly, accurately depicting the content in prescribed text descriptions. Here we tackle this problem with a two-stage approach: text2length sampling and text2motion generation. Text2length involves sampling from the learned distribution function of motion lengths conditioned on the input text. This is followed by our text2motion module using temporal variational autoen-coder to synthesize a diverse set of human motions of the sampled lengths. Instead of directly engaging with pose sequences, we propose motion snippet code as our internal motion representation, which captures local semantic motion contexts and is empirically shown to facilitate the generation of plausible motions faithful to the input text. Moreover, a large-scale dataset of scripted 3D Human motions, HumanML3D, is constructed, consisting of 14,616 motion clips and 44,970 text descriptions.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/4ccb30b632d847764591481bde34613b69530692

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “[13] encourage diverse generation using VAE based models [22], yet not generalizing outside of the limited available data.”
A Survey on Generative Diffusion Models (2022, methodology): “Graph Graph-based neural networks are gaining popularity due to their expressiveness in human pose [127], molecules [134], and proteins [135] [136].”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “The study primarily focuses on two text-to-motion datasets: HumanML3D [10] and KIT [31].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “Ablation study of different dilation rate in VQ-VAE on HumanML3D [1] test set.”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “HumanML3D [48] re-annotates the AMASS dataset [73] and the HumanAct12 dataset [1].”
Human Motion Diffusion as a Generative Prior (2023, methodology): “To provide a proper analysis, we generate a 32-intervals long sequence, then apply HumanML3D metrics on the intervals themselves, and once again for the transition.”
EDGE: Editable Dance Generation From Music (2022, background): “Indeed, in the text-to-motion domain, where significantly more paired data is available [14], the standard feature extractors for FID-style metrics depend on deep contrastively learned models [53, 58] Our experiments suggest that the current FID metrics can be further improved for evaluating motion ”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “Two 3D human motion-language datasets are considered for evaluation: – HumanML3D [14] is a large 3D human motion dataset that covers a broad range of human actions such as locomotion, sports, and dancing.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “In terms of text-to-motion generation, T2M [15] extended the temporal VAE to learn the probabilistic mapping between texts and motions.”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “We evaluate our proposed ReMoDiffuse on two standard text-to-motion generation benchmarks, HumanML3D [7] and KIT-ML [17].”
Human Motion Diffusion Model (2022, methodology): “This condition can be any real-world signal that will dictate the synthesis, such as audio (Li et al., 2021; Aristidou et al., 2022), natural language (text-to-motion) (Tevet et al., 2022; Guo et al., 2022a) or a discrete class (action-to-motion) (Guo et al., 2020; Petrovich et al., 2021).”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “In Table 1, we compare our PhysDiff model to the SOTA methods: JL2P [1], Text2Gesture [9], T2M [22] and MDM [80].”