ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model

By Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, Ziwei Liu. IEEE International Conference on Computer Vision, 2023

333 citations recorded by the source corpus.

3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process. ReMoDiffuse enhances the generalizability and diversity of text-driven motion generation with three key designs: 1) Hybrid Retrieval finds appropriate references from the database in terms of both semantic and kinematic similarities. 2) Semantic-Modulated Transformer selectively absorbs retrieval knowledge, adapting to the difference between retrieved samples and the target motion sequence. 3) Condition Mixture better utilizes the retrieval database during inference, overcoming the scale sensitivity in classifier-free guidance. Extensive experiments demonstrate that ReMoDiffuse outperforms state-of-the-art methods by balancing both text-motion consistency and motion quality, especially for more diverse motion generation. Project page: https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/731ac2fd7a5586d17cb8959c190158d429821a60

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Qi Tian
3. Lingxi Xie
4. Wenyu Liu
5. Xiaopeng Zhang
6. Alan L. Yuille
7. Yi Yang
8. Luc Van Gool

In Control and Systems Engineering

1. Chuan Guo
2. Li Cheng
3. Sen Wang
4. Qi Tian
5. Yuxuan Mu
6. Muhammad Gohar Javed
7. Xinggang Wang
8. Lingxi Xie

Citations in the corpus

• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, positive): “Due to their powerful capacities, retrieval techniques have been successfully incorporated into advanced generative models in the era of AI-Generated Content (AIGC) [68, 112, 134].”
• Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “To fit in the input length limit, it reserves the contexts that are most relevant to the query.”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “We compare our approach to a set of existing state-of-the-art works ranging from VAE [15], diffusion-based models [9, 42, 51], to autoregressive models [16, 49].”
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “In addition to generating 3D assets from text, there are methods where 3D diffusion models [1, 7, 10, 26, 55, 63, 72, 88, 89, 91] generate motion sequences based on text-motion data.”