Action2Motion: Conditioned Generation of 3D Human Motions

By Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, Li Cheng. ACM Multimedia, 2020

642 citations recorded by the source corpus.

Action recognition is a relatively established task, where given an input sequence of human motion, the goal is to predict its action category. This paper, on the other hand, considers a relatively new problem, which could be thought of as an inverse of action recognition: given a prescribed action type, we aim to generate plausible human motion sequences in 3D. Importantly, the set of generated motions are expected to maintain its diversity to be able to explore the entire action-conditioned motion space; meanwhile, each sampled sequence faithfully resembles a natural human body articulation dynamics. Motivated by these objectives, we follow the physics law of human kinematics by adopting the Lie Algebra theory to represent the natural human motions; we also propose a temporal Variational Auto-Encoder (VAE) that encourages a diverse sampling of the motion space. A new 3D human motion dataset, HumanAct12, is also constructed. Empirical experiments over three distinct human motion datasets (including ours) demonstrate the effectiveness of our approach.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/b7b4996858b91bc4db5dd5ce13215b5422591529

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, methodology): “For experiments on NTU-13 and HumanAct12, we directly use the provided recognition models of [20] that operate on joint coordinates.”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, background): “Hu-manML3D is a scripted 3D human motion dataset that originates from and textually reannotates the HumanAct12 [9] and AMASS datasets [12].”
BABEL: Bodies, Action and Behavior with English Labels (2021, background): “, motion synthesis conditioned on actions [12], or sentences [3, 19].”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “Human Motion Synthesis involves generating diverse and realistic human-like motion using multi-modal inputs, such as text [10, 30, 58, 46, 11, 1, 17], action [29, 12, 46, 52], and incomplete motion [56, 61, 24, 46].”
Generating Diverse and Natural 3D Human Motions from Text (2022, methodology): “Evaluation Metrics from [12] are adopted here, which include Frechet Inception Distance (FID), diversity and multimodality.”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “However, the literature in action-conditioned motion generation task [1], [2], [3] argue that this metric should be close to the real motion.”
EDGE: Editable Dance Generation From Music (2022, background): “joystick control [31], class-conditioning [15, 40], text-tomotion [41, 58], seed motions [8, 21, 43, 57]—is also an active area of study.”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “ACTOR [Petrovich et al. 2021] and Action2Motion [Guo et al. 2020] suggested learning this multi-modal distribution from existing action recognition datasets using Conditional Variational-Autoencoder(CVAE) [Sohn et al. 2015] architectures.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “The motion sequences are originally from AMASS [45] and HumanAct12 [23] but with specific pre-processing: motion is scaled to 20 FPS; those that are longer than 10 seconds are randomly cropped to 10-second”
TEMOS: Generating diverse human motions from textual descriptions (2022, methodology): “On the other hand, the conditioned synthesis can be further divided into two categories: deterministic [2, 13, 29] or probabilistic [1, 15, 26, 27, 30, 39].”
Human Motion Diffusion Model (2022, methodology): “We use the metrics they suggest (FID, KID, precision/recall and multimodality), and run on an unconstrained version of the HumanAct12 (Guo et al., 2020) dataset.”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “Action category based human motion generation also draws considerable interests by resorting to a diverse range of learning strategies, including GANs [52], VAEs [16,15], Transformers [35] and GCNs [57].”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “Tables 2 and 3 summarize the results on HumanAct12 and UESTC, respectively, where we compare PhysDiff against the SOTA methods: MDM [80], INR [10], Action2Motion [23], and ACTOR [61].”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Lastly, we introduce action-tomotion [46, 19], a reverse problem of the classical action recognition task.”