MotionCLIP: Exposing Human Motion Generation to CLIP Space

By Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, Daniel Cohen-Or. European Conference on Computer Vision, 2022

555 citations recorded by the source corpus.

We introduce MotionCLIP, a 3D human motion auto-encoder featuring a latent embedding that is disentangled, well behaved, and supports highly semantic textual descriptions. MotionCLIP gains its unique power by aligning its latent space with that of the Contrastive Language-Image Pre-training (CLIP) model. Aligning the human motion manifold to CLIP space implicitly infuses the extremely rich semantic knowledge of CLIP into the manifold. In particular, it helps continuity by placing semantically similar motions close to one another, and disentanglement, which is inherited from the CLIP-space structure. MotionCLIP comprises a transformer-based motion auto-encoder, trained to reconstruct motion while being aligned to its text label's position in CLIP-space. We further leverage CLIP's unique visual understanding and inject an even stronger signal through aligning motion to rendered frames in a self-supervised manner. We show that although CLIP has never seen the motion domain, MotionCLIP offers unprecedented text-to-motion abilities, allowing out-of-domain actions, disentangled editing, and abstract language specification. For example, the text prompt"couch"is decoded into a sitting down motion, due to lingual similarity, and the prompt"Spiderman"results in a web-swinging-like solution that is far from seen during training. In addition, we show how the introduced latent space can be leveraged for motion interpolation, editing and recognition.

View Paper: https://www.semanticscholar.org/paper/e82df4b6a3628501fce67835ad8316d6525ad133

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yuval Alaluf
2. Chuan Guo
3. Rinon Gal
4. Biao Jiang
5. Daniel Cohen-Or
6. Mingyuan Zhang
7. Or Patashnik
8. Xin Chen

Citations in the corpus

• Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “To this end, many works propose to learn a joint embedding for language and motion using auto-encoders [3, 21, 65] and VAEs [52, 53].”
• MotionGPT: Human Motion as a Foreign Language (2023, unknown): “MotionCLIP [45] utilizes the language and visual understanding of CLIP [33] to align its latent space with a motion auto-encoder.”
• Executing your Commands via Motion Diffusion in Latent Space (2022, background): “C V ] 19 M a y 2023 motion sequences are quite different, it is not easy to learn a probabilistic mapping function from the textural descriptors to the motion sequences, which is also mentioned in the previous work, MotionCLIP [68].”
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, background): “MotionCLIP [23] attempts to enhance the generalizability of text-to-motion generation.”
• Human Motion Diffusion Model (2022, background): “This condition can be any real-world signal that will dictate the synthesis, such as audio (Li et al., 2021; Aristidou et al., 2022), natural language (text-to-motion) (Tevet et al., 2022; Guo et al., 2022a) or a discrete class (action-to-motion) (Guo et al., 2020; Petrovich et al., 2021).”
• Human Motion Diffusion as a Generative Prior (2023, background): “…generation approach [Ho et al. 2020; Sohl-Dickstein et al. 2015] have quickly found their way into the field, yielding motion generation models that produce diverse and high-quality sequences from text or other forms of control [Guo et al. 2022; Petrovich et al. 2022; Tevet et al. 2022, 2023].”
• MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “This architecture is also employed in later works (Tevet et al., 2022; Hong et al., 2022; Petrovich et al., 2022).”
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “, 2022), motion synthesis (Tevet et al., 2022; Petrovich et al., 2022), style transfer (Kwon & Ye, 2021; Liu et al.”
• Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, background): “Text-to-3D with 2D Supervision Recently, the success of text-guided synthesis in numerous domains [1, 2, 17, 34, 44], has motivated a surge of works that use Language-Image models to guide 3D scenes representations.”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “MotionCLIP [41] and ohMG [28] model text-to-motion in an unsupervised manner using the large pretrained CLIP [39] model.”