Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory

By Lian Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, Ziwei Liu. Computer Vision and Pattern Recognition, 2022

316 citations recorded by the source corpus.

Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated dance sequence also needs to maintain temporal coherency with different music genres. To tackle these challenges, we propose a novel music-to-dance framework, Bailando, with two powerful components: 1) a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequence to a quantized codebook, 2) an actor-critic Generative Pre-trained Transformer (GPT) that composes these units to a fluent dance coherent to the music. With the learned choreographic memory, dance generation is realized on the quantized units that meet high choreography standards, such that the generated dancing sequences are confined within the spatial constraints. To achieve synchronized alignment between diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a newly-designed beat-align reward function. Extensive experiments on the standard benchmark demonstrate that our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively. Notably, the learned choreographic memory is shown to discover human-interpretable dancing-style poses in an unsupervised manner. Code and video demo are available at https://github.com/lisiyao21/Bailando/

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/45b0f30ee83f324ccdffd608818ffb2d50de4a8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yuxuan Mu
2. Muhammad Gohar Javed
3. Sen Wang
4. Li Cheng
5. Chuan Guo
6. Xiaodong Cun
7. Yong Zhang
8. Jo-Han Tseng

In Control and Systems Engineering

1. Chuan Guo
2. Li Cheng
3. Sen Wang
4. Jo-Han Tseng
5. Ying Shan
6. Fei Wang
7. Yuxuan Mu
8. Rodrigo Castellon

Citations in the corpus

• Generating Human Motion from Textual Descriptions with Discrete Representations (2023, background): “Motion can also be generated with music to produce 3D dance motion [6,13,37,38,40,41].”
• EDGE: Editable Dance Generation From Music (2022, methodology): “Later works instead synthesize motion from scratch by training on large datasets [1, 10, 22, 24, 28, 30, 42, 49, 54] and propose many modeling approaches, including adversarial learning, recurrent neural networks, and transformers.”
• MotionGPT: Human Motion as a Foreign Language (2023, unknown): “To represent motion in discrete tokens, we pre-train a 3D human motion tokenizer V based on the Vector Quantized Variational Autoencoders (VQ-VAE) architecture used in [48, 42, 11, 57].”
• MoMask: Generative Masked Modeling of 3D Human Motions (2023, background): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”
• SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, methodology): “For the alignment of the audio and generated head motions, we compute Beat Align Score as in Bailando [21].”