Generating Human Motion from Textual Descriptions with Discrete Representations

By Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xiaodong Shen. Computer Vision and Pattern Recognition, 2023

716 citations recorded by the source corpus.

In this work, we investigate a simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer (GPT) for human motion generation from textural descriptions. We show that a simple CNN-based VQ-VAE with commonly used training recipes (EMA and Code Reset) allows us to obtain high-quality discrete representations. For GPT, we incorporate a simple corruption strategy during the training to alleviate training-testing discrepancy. Despite its simplicity, our T2M-GPT shows better performance than competitive approaches, including recent diffusion-based approaches. For example, on HumanML3D, which is currently the largest dataset, we achieve comparable performance on the consistency between text and generated motion (R-Precision), but with FID 0.116 largely outperforming MotionDiffuse of 0.630. Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach. Our work suggests that VQ-VAE still remains a competitive approach for human motion generation. Our implementation is available on the project page: https://mael-zys.github.io/T2M-GPT/.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/99bd96961a880adad88d087c9dc45f6cfbae1ea6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Li Cheng
2. Sen Wang
3. Yuxuan Mu
4. Muhammad Gohar Javed
5. Chuan Guo
6. Liang Pan
7. Mingyuan Zhang
8. Fangzhou Hong

In Control and Systems Engineering

1. Chuan Guo
2. Li Cheng
3. Sen Wang
4. Fangzhou Hong
5. Yuxuan Mu
6. Ziwei Liu
7. Mingyuan Zhang
8. Lei Yang

Citations in the corpus

• MotionGPT: Human Motion as a Foreign Language (2023, neutral): “079 T2M-GPT [57] 0 .”
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “Table 1 and Table 2 show the comparison between our proposed ReMoDiffuse and four other existing works, including recent diffusion models-based algorithms [24, 27], one VAE-based generative model [7], and one GPT-style generative model [26].”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “T2M-GPT [49] further enhances the performance using EMA and code reset techniques.”