MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model

By Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, Ziwei Liu. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

1,020 citations recorded by the source corpus.

Human motion modeling is important for many modern graphics applications, which typically require professional skills. In order to remove the skill barriers for laymen, recent motion generation methods can directly generate human motions conditioned on natural languages. However, it remains challenging to achieve diverse and fine-grained motion generation with various text inputs. To address this problem, we propose MotionDiffuse, one of the first diffusion model-based text-driven motion generation frameworks, which demonstrates several desired properties over existing methods. 1) Probabilistic Mapping. Instead of a deterministic language-motion mapping, MotionDiffuse generates motions through a series of denoising steps in which variations are injected. 2) Realistic Synthesis. MotionDiffuse excels at modeling complicated data distribution and generating vivid motion sequences. 3) Multi-Level Manipulation. MotionDiffuse responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts. Our experiments show MotionDiffuse outperforms existing SoTA methods by convincing margins on text-driven motion generation and action-conditioned motion generation. A qualitative analysis further demonstrates MotionDiffuse's controllability for comprehensive motion generation.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/1bbf99b5bfe9869876ac3bdd2999e16b2632c283

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Human Motion Diffusion as a Generative Prior (2023, methodology): “Very recently, MDM [Tevet et al. 2023], MotionDiffuse [Zhang et al. 2022], MoFusion [Dabral et al. 2023], and FLAME [Kim et al. 2022] successfully implemented motion generation neural models using the Denoising Diffusion Probabilistic Models (DDPM) [Ho et al. 2020] setting, which was originally…”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Among all conditional modalities, text-based conditional human motion synthesis has been driving and dominating research frontiers because the language descriptors provide a convenient and natural user interface for people to interact with computers [47, 2, 75, 69, 28].”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “Recently, diffusion models have been introduced into motion generation [27, 24].”
Human Motion Diffusion Model (2022, background): “More recently, concurrent to this work, Zhang et al. (2022) and Kim et al. (2022) have suggested diffusion models for motion generation.”
EDGE: Editable Dance Generation From Music (2022, methodology): “Most recently, diffusion-based methods have demonstrated strong performance in generating motions conditioned on text [26,58,63].”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “With flexible conditioning, it is further extended to the language-conditioned image [39, 49, 51], video [18, 54], and 3D generation [42, 61, 78].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “Note that MDM [66] and MotionDiffuse [74] evaluate their models with the ground-truth motion length, which is not practical for real applications.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Multi-Modal Learning Text-to-Image Generation [7],[216],[231],[189],[87],[65],[229],[126],[269],[9],[299],[321] Scene Graph-to-Image Generation [306] Text-to-3D Generation [296],[155],[207] Text-to-Motion Generation [263, 322],[130] Text-to-Video Generation [245],[96],[283],[210] Text-to-Audio Gener”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “In Table 1, we compare our method to the SOTA methods: JL2P [1], Text2Gesture [9], T2M [22], MotionDiffuse [94], and MDM [79].”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, background): “In diffusion meth-ods, a network is learned to gradually denoise the motion sequence, supervised by a scheduled diffusion process [9, 22, 23, 30, 42, 43, 50].”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “…is one of the most important motion generation tasks, due to the user-2 Methods Text-to-Motion Motion-to-Text Motion Prediction Motion In-between Random Motion Random Description T2M-GPT [57] MLD [52] TM2T [11] MDM [46] MotionDiffuse[58] MotionGPT (Ours) friendly and convenient language input.”