AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

By Ruilong Li, Sha Yang, David A. Ross, Angjoo Kanazawa. IEEE International Conference on Computer Vision, 2021

734 citations recorded by the source corpus.

We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses—the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict N future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively. The code and the dataset can be found at: https://google.github.io/aichoreographer.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/021bd56b6c95d0196f3b8e818c948ea0702b0836

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, methodology): “Very recently, [39] presents a Transformer-based method for dance generation conditioned on music and past motion.”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, background): “Motion can also be generated with music to produce 3D dance motion [6,13,36,37,39,40].”
Multimodal Learning With Transformers: A Survey (2022, background): “Depth to Binaural Audio [14], music piece & seed 3D motion to long-range future 3D motions [146], X-raying image & question to answer [244], video & text & audio to text [245]), and”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “These methods often use GANs or VAEs to generate motions from various conditions such as past motions [3, 7, 87, 91, 92], key frames [24], music [37, 38, 100], text [1, 9, 22, 62], and action labels [10, 23, 61].”
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, neutral): “More recent methods [36, 41, 49, 58] use attention [59].”
Human Motion Diffusion Model (2022, background): “This condition can be any real-world signal that will dictate the synthesis, such as audio (Li et al., 2021; Aristidou et al., 2022), natural language (text-to-motion) (Tevet et al., 2022; Guo et al., 2022a) or a discrete class (action-to-motion) (Guo et al., 2020; Petrovich et al., 2021).”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “, 2018), 3D keypoints (Ionescu et al., 2013; Joo et al., 2015; Mehta et al., 2017; Trumble et al., 2017; Li et al., 2021) and statistical model parameters (Yu et al.”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “To project the motion manifold into the latent space, we learn a transformer-based auto-encoder [Vaswani et al. 2017], adapted to the motion domain [Petrovich et al. 2021; Wang et al. 2021b; Li et al. 2021].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Transformers have also been widely used in video forecasting problems like action anticipation [28], trajectory estimation [30,90] and human motion prediction [45,66].”
EDGE: Editable Dance Generation From Music (2022, methodology): “Dataset In this work, we use AIST++ [30], a dataset consisting of 1,408 high-quality dance motions paired to music from a diverse set of genres.”
Executing your Commands via Motion Diffusion in Latent Space (2022, background): “Human Motion Synthesis allows rich inputs of multimodal data, such as text [16, 47, 68, 18, 2, 28], action category [46, 19], incomplete pose sequences [12, 20, 68], control signals [65, 66, 45], musics [34, 33, 32] and image(s) [55, 9], here, we focus on some typical tasks.”
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory (2022, methodology): “To further understand the real visual performance of our method, we conduct a user study among the dance sequences generated by each compared method and the ground truth data in AIST++ test set.”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “Even in AIST++, almost half of the generated sound videos can fool users although the ﬁne-grained parts of persons are difﬁcult to be well generated.”
TEMOS: Generating diverse human motions from textual descriptions (2022, background): “Motion synthesis methods can be broadly divided into two categories: (i) unconstrained generation, which models the entire space of possible motions [51, 56, 58] and (ii) conditioned synthesis, which aims for controllability such as using music [26, 27, 28], speech [7, 14], action [15, 39], and text”