MotionGPT: Human Motion as a Foreign Language

By Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, Tao Chen. Neural Information Processing Systems, 2023

638 citations recorded by the source corpus.

Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this"motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d212fa27f5868f0fd106e1a7bba908fd47da0816

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiangmiao Pang
2. Dahua Lin
3. Sen Wang
4. Chuan Guo
5. Tai Wang
6. Yilun Chen
7. Li Cheng
8. Chen Change Loy

In Computer Vision and Pattern Recognition

1. Jiangmiao Pang
2. Li Cheng
3. Dahua Lin
4. Sen Wang
5. Yuxuan Mu
6. Muhammad Gohar Javed
7. Chuan Guo
8. Tai Wang

Citations in the corpus

• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…M-LLMs for video data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models…”
• PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “They aim at having the capability to interpret diverse modalities, including but not limited to images [13, 18, 27, 45, 56], audio [17], motion [21], etc.”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Among them, it has become popular to engage generative transformers in modeling human motions [13, 16, 21, 49].”