Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

By Mathis Petrovich, Michael J. Black, Gul Varol. IEEE International Conference on Computer Vision, 2021

716 citations recorded by the source corpus.

We tackle the problem of action-conditioned generation of realistic and diverse human motion sequences. In contrast to methods that complete, or extend, motion sequences, this task does not require an initial pose or sequence. Here we learn an action-aware latent representation for human motions by training a generative variational autoencoder (VAE). By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action. Specifically, we design a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets. We evaluate our approach on the NTU RGB+D, HumanAct12 and UESTC datasets and show improvements over the state of the art. Furthermore, we present two use cases: improving action recognition through adding our synthesized data to training, and motion denoising. Code and models are available on our project page [53].

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/4d44de6653f6192efba8aef36f814d4c0420f21d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, background): “However, even recent methods [38, 46, 61, 64], mostly study the body “in isolation”; i.e., with no scene context.”
TEMOS: Generating diverse human motions from textual descriptions (2022, methodology): “Such skinned generations were considered in other work on unconstrained or action-conditioned generation [39, 57].”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Lastly, we introduce action-tomotion [46, 19], a reverse problem of the classical action recognition task.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, background): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, background): “Some of the most recent works on 3D body motion synthesis [1, 4, 46] and 3D dance generation [36, 37, 57] have explored the power of transformer in modeling sequential data and produced impressive results.”
EDGE: Editable Dance Generation From Music (2022, background): “Auxiliary losses are commonly used in kinematic motion generation settings to improve physical realism in the absence of true simulation [40, 48, 52].”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “Unlike Ac-tion2Motion, ACTOR embeds the whole motion sequence into the latent space.”
Human Motion Diffusion Model (2022, methodology): “For both datasets we use the sequences provided by Petrovich et al. (2021).”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “Action category based human motion generation also draws considerable interests by resorting to a diverse range of learning strategies, including GANs [52], VAEs [16,15], Transformers [35] and GCNs [57].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “ACTOR [51] proposes a transformer-based VAE to generate motion in a non-autoregressive fashion from a pre-defined action class.”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “These methods often use GANs or VAEs to generate motions from various conditions such as past motions [3, 7, 87, 91, 92], key frames [24], music [37, 38, 100], text [1, 9, 22, 62], and action labels [10, 23, 61].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Transformers have also been widely used in video forecasting problems like action anticipation [28], trajectory estimation [30,90] and human motion prediction [45,66].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “The mesh vertices locations 𝑣 1: 𝑇 are calculated according to SMPL specifications with 𝛽 = 0 and a neutral-gender body model following Petrovich et al. [Petrovich et al. 2021].”
Generating Diverse and Natural 3D Human Motions from Text (2022, background): “…extend the partial motion sequence with newly generated poses; the work of [45] instead models the spatiotemporal structures of human dynamics with a GCN-based GAN; mean-while, VAE modelling and transformer architecture are promoted by [11, 12, 29] to incorporate temporal dependencies.”
MotionGPT: Human Motion as a Foreign Language (2023, neutral): “Previous research on human motion has explored various tasks, including motion generation [29, 10, 46, 52, 57], motion captioning [9, 11], and motion prediction [56, 61, 24].”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “As for the motion encoder, we use a 4-layer ACTOR [15] Encoder.”