TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

By Chuan Guo, Xinxin Zuo, Sen Wang, Li Cheng. European Conference on Computer Vision, 2022

433 citations recorded by the source corpus.

Inspired by the strong ties between vision and language, the two intimate human sensing and communication modalities, our paper aims to explore the generation of 3D human full-body motions from texts, as well as its reciprocal task, shorthanded for text2motion and motion2text, respectively. To tackle the existing challenges, especially to enable the generation of multiple distinct motions from the same text, and to avoid the undesirable production of trivial motionless pose sequences, we propose the use of motion token, a discrete and compact motion representation. This provides one level playing ground when considering both motions and text signals, as the motion and text tokens, respectively. Moreover, our motion2text module is integrated into the inverse alignment process of our text2motion training pipeline, where a significant deviation of synthesized text from the input text would be penalized by a large training loss; empirically this is shown to effectively improve performance. Finally, the mappings in-between the two modalities of motions and texts are facilitated by adapting the neural model for machine translation (NMT) to our context. This autoregressive modeling of the distribution over discrete motion tokens further enables non-deterministic production of pose sequences, of variable lengths, from an input text. Our approach is flexible, could be used for both text2motion and motion2text tasks. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach on both tasks over a variety of state-of-the-art methods. Project page: https://ericguo5513.github.io/TM2T/

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/fb43ae5748c5a8cbbe2b2b8b7efa01e13888f870

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Gang Yu
2. Luc Van Gool
3. Li Cheng
4. Zilong Huang
5. Jingyi Yu
6. Ziwei Liu
7. Sen Wang
8. Tao Chen

In Control and Systems Engineering

1. Tao Chen
2. Gang Yu
3. Wen Liu
4. Xin Chen
5. Jingyi Yu
6. Biao Jiang
7. Chuan Guo
8. Li Cheng

In Human-Computer Interaction

1. Gang Yu
2. Ziwei Liu
3. Jingyi Yu
4. Shaoli Huang

Citations in the corpus

• MotionGPT: Human Motion as a Foreign Language (2023, positive): “…is one of the most important motion generation tasks, due to the user-2 Methods Text-to-Motion Motion-to-Text Motion Prediction Motion In-between Random Motion Random Description T2M-GPT [57] MLD [52] TM2T [11] MDM [46] MotionDiffuse[58] MotionGPT (Ours) friendly and convenient language input.”
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, background): “TM2T [8] regards the text-driven motion generation task as a translation task between natural languages and motion sequences.”
• Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “TM2T [23] considers text-to-motion and motion-to-text tasks.”
• Executing your Commands via Motion Diffusion in Latent Space (2022, background): “Human Motion Synthesis allows rich inputs of multimodal data, such as text [16, 47, 69, 18, 2, 28], action category [46, 19], incomplete pose sequences [12, 20, 69], control signals [65, 66, 45], musics [34, 33, 32] and image(s) [55, 9], here, we focus on some typical tasks.”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”