Human Motion Diffusion Model

By Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, Amit H. Bermano. International Conference on Learning Representations, 2022

1,358 citations recorded by the source corpus.

Natural and expressive human motion generation is the holy grail of computer animation. It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain. MDM is transformer-based, combining insights from motion generation literature. A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion step. This facilitates the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss. As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion and action-to-motion. https://guytevet.github.io/mdm-page/ .

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/15736f7c205d961c00378a938daffaacb5a0718d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

EDGE: Editable Dance Generation From Music (2022, methodology): “Editing To enable editing for dances generated by EDGE, we use the standard masked denoising technique from diffusion image inpainting [36], and more recently text-to-motion models [26,58].”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “Diffusion models for image synthesis Diffusion models (DMs) [51, 53] can synthesize detailed media in many formats, such as images [34, 38], 3d shapes [66] and animations [54].”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “The 3D diffusion models used in our method are Shap-E [24] and MDM [72], and we load the Shap-E model finetuned on Objaverse [11] in Cap3D [42].”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “Therefore, several motion diffusion models [77, 92, 65] have been proposed which demonstrate SOTA motion generation performance.”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “…is one of the most important motion generation tasks, due to the user-2 Methods Text-to-Motion Motion-to-Text Motion Prediction Motion In-between Random Motion Random Description T2M-GPT [57] MLD [52] TM2T [11] MDM [46] MotionDiffuse[58] MotionGPT (Ours) friendly and convenient language input.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “With flexible conditioning, it is further extended to the language-conditioned image [41, 51, 53], video [18, 56], and 3D generation [44, 63, 85].”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Among all conditional modalities, text-based conditional human motion synthesis has been driving and dominating research frontiers because the language descriptors provide a convenient and natural user interface for people to interact with computers [47, 2, 75, 69, 28].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Multi-Modal Learning Text-to-Image Generation [7],[216],[231],[189],[87],[65],[229],[126],[269],[9],[299],[321] Scene Graph-to-Image Generation [306] Text-to-3D Generation [296],[155],[207] Text-to-Motion Generation [263, 322],[130] Text-to-Video Generation [245],[96],[283],[210] Text-to-Audio Gener”
ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, methodology): “Other published generative AI models are able to generate human motion [31] or, in the case of ChatBCG, slides using ChatGPT as a surrogate model.”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “…of animations available in O BJAVERSE can support a wide range of research in temporal 3D learning, such as building text-based animation generative models [71], representing object changes over time with NERFs [51,54], and temporal self-supervised learning via. future frame prediction [28,82].”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “…in various domains, including images (Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022), video (Ho et al., 2022; Singer et al., 2022), 3D scenes (M ´ uller et al., 2022), and motion sequences (Yuan et al., 2022; Tevet et al., 2022).”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “Following MDM [24], we predict the clean state x 0 .”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “Concurrent with this work, Tevet et al. [44], and Kim et al. [45] also attempted to exploit the potential of diffusion models on motion generation.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “This user study involves 42 AMT users with master recognition, with the side-by-side comparisons between MoMask and each of the state-of-the-art methods including MDM [42], MLD [9] and T2M-GPT [49].”
Human Motion Diffusion as a Generative Prior (2023, methodology): “Very recently, MDM [Tevet et al. 2022b], MotionDiffuse [Zhang et al.”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, methodology): “Fig.4 also shows our controller’s ability to imitate reference motion generated directly from a motion language model MDM [41].”