Human Motion Diffusion as a Generative Prior

By Yonatan Shafir, Guy Tevet, Roy Kapon, Amit H. Bermano. International Conference on Learning Representations, 2023

406 citations recorded by the source corpus.

Recent work has demonstrated the significant potential of denoising diffusion models for generating human motion, including text-to-motion capabilities. However, these methods are restricted by the paucity of annotated motion data, a focus on single-person motions, and a lack of detailed control. In this paper, we introduce three forms of composition based on diffusion priors: sequential, parallel, and model composition. Using sequential composition, we tackle the challenge of long sequence generation. We introduce DoubleTake, an inference-time method with which we generate long animations consisting of sequences of prompted intervals and their transitions, using a prior trained only for short clips. Using parallel composition, we show promising steps toward two-person generation. Beginning with two fixed priors as well as a few two-person training examples, we learn a slim communication block, ComMDM, to coordinate interaction between the two resulting motions. Lastly, using model composition, we first train individual priors to complete motions that realize a prescribed motion for a given joint. We then introduce DiffusionBlending, an interpolation mechanism to effectively blend several such models to enable flexible and efficient fine-grained joint and trajectory-level control and editing. We evaluate the composition methods using an off-the-shelf motion diffusion model, and further compare the results to dedicated models trained for these specific tasks.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/cb89947993021a3ee3c9f2bb926f6cab71d39793

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Qi Tian
3. Lingxi Xie
4. Wenyu Liu
5. Luc Van Gool
6. Alan L. Yuille
7. Yi Yang
8. Dacheng Tao

In Control and Systems Engineering

1. Tao Chen
2. Qi Tian
3. Gang Yu
4. Wen Liu
5. Xin Chen
6. Jingyi Yu
7. Biao Jiang
8. Xinggang Wang

Citations in the corpus

• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “In addition to generating 3D assets from text, there are methods where 3D diffusion models [1, 7, 10, 26, 55, 63, 72, 88, 89, 91] generate motion sequences based on text-motion data.”
• MotionGPT: Human Motion as a Foreign Language (2023, positive): “It is interesting to model the human interaction scenarios in a motion-language framework and generate controllable motions [41].”