HP-GAN: Probabilistic 3D Human Motion Prediction via GAN

By Emad Barsoum, John R. Kender, Zicheng Liu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017

375 citations recorded by the source corpus.

Predicting and understanding human motion dynamics has many applications, such as motion synthesis, augmented reality, security, and autonomous vehicles. Due to the recent success of generative adversarial networks (GAN), there has been much interest in probabilistic estimation and synthetic data generation using deep neural network architectures and learning algorithms. We propose a novel sequence-to-sequence model for probabilistic human motion prediction, trained with a modified version of improved Wasserstein generative adversarial networks (WGAN-GP), in which we use a custom loss function designed for human motion prediction. Our model, which we call HP-GAN, learns a probability density function of future human poses conditioned on previous poses. It predicts multiple sequences of possible future human poses, each from the same input sequence but a different vector z drawn from a random distribution. Furthermore, to quantify the quality of the non-deterministic predictions, we simultaneously train a motion-quality-assessment model that learns the probability that a given skeleton sequence is a real human motion. We test our algorithm on two of the largest skeleton datasets: NTURGB-D and Human3.6M. We train our model on both single and multiple action types. Its predictive power for long-term motion estimation is demonstrated by generating multiple plausible futures of more than 30 frames from just 10 frames of input. We show that most sequences generated from the same input have more than 50% probabilities of being judged as a real human sequence. We published all the code used in this paper to https://github.com/ebarsoum/hpgan.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/d488ceb4ea85d4d31a933a16b0abd0af042d5b98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, background): “The motion prediction helps machines understand human behaviors, attracting considerable attention [9, 20, 32, 5, 12, 2].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “Approaches mainly focus on efficiently and effectively fusing spatial and temporal information to generate deterministic future motion through different models: RNN [12,20,50,51], GAN [9,30], GCN [49], Attention [48] or even simply MLP [11, 17, 25].”
Mode-adaptive neural networks for quadruped motion control (2018, neutral): “This has been explored in the physically-based environment [Merel et al. 2017b] and preliminary results for motion capture data [Barsoum et al. 2017], but a framework for interactive character control is yet to be done.”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “HP-GAN [34] attempts to supervise the motion prediction results without the speciﬁc ground truth.”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, background): “Examples include HP-GAN [6] and recurrent VAE [22] for future motion prediction.”
Hierarchical Style-based Networks for Motion Synthesis (2020, methodology): “For long-range generation, we compare our model with temporal prediction baselines: HP-GAN [1] and MTVAE [39].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, methodology): “he multi-modal distribution of future human motion. Deep generative models, e.g., variational autoencoders (VAEs) [38], are effective tools to model multi-modal data distributions. Most existing work [68, 46,6,61,42,71,3] using deep generative models for human motion prediction is focused on the des”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “These methods often use GANs or VAEs to generate motions from various conditions such as past motions [7, 83, 3, 88, 87], key frames [24], music [37, 96, 36], text [1, 9, 22, 61], and action labels [23, 60, 10].”
Robust motion in-betweening (2020, methodology): “Remaining ambiguity can be alleviated with generative adversarial training [Goodfellow et al. 2014], in which the motion generator learns to fool an additional discriminator network that tries to differentiate generated sequences from real sequences.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, background): “Research in motion modelling has shown that combining sequential architectures and adversarial training can be used to predict future motion sequences based on previous ones [9, 20] or to generate human motion sequences [2].”
MotionDiffuser: Controllable Multi-Agent Motion Prediction Using Diffusion (2023, methodology): “HP-GAN [1] learns a probability density function (PDF) of future human poses conditioned on previous poses using an improved Wasserstein Generative Adversarial Network (GAN).”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “In contrast, present work on human motion synthesis typically represents behavior directly by means of the observed sequence of postures [3, 40, 63, 57].”