On Human Motion Prediction Using Recurrent Neural Networks

By Julieta Martinez, Michael J. Black, Javier Romero. Computer Vision and Pattern Recognition, 2017

1,057 citations recorded by the source corpus.

Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality. Following the success of deep learning methods in several computer vision tasks, recent work has focused on using deep recurrent neural networks (RNNs) to model human motion, with the goal of learning time-dependent representations that perform tasks such as short-term motion prediction and long-term human motion synthesis. We examine recent work, with a focus on the evaluation methodologies commonly used in the literature, and show that, surprisingly, state of the art performance can be achieved by a simple baseline that does not attempt to model motion at all. We investigate this result, and analyze recent RNN methods by looking at the architectures, loss functions, and training procedures used in state-of-the-art approaches. We propose three changes to the standard RNN models typically used for human motion, which results in a simple and scalable RNN architecture that obtains state-of-the-art performance on human motion prediction.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7f065002afcb90240a41f05f138269c5675b9805

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

TEMOS: Generating diverse human motions from textual descriptions (2022, background): “These approaches may suffer from drift over time and often, eventually, produce static poses [37].”
Behavior-Driven Synthesis of Human Dynamics (2021, methodology): “[40] use residual RNN architectures to directly model motion velocities.”
Character controllers using motion VAEs (2020, background): “RNNs have been extensively explored for short and long-term human motion prediction in computer vision [Fragkiadaki et al. 2015; Martinez et al. 2017].”
History Repeats Itself: Human Motion Prediction via Motion Attention (2020, methodology): “As such, they have been widely employed for human motion prediction [5,11,22,7].”
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints (2018, methodology): “Once the assignment has been resolved between every pair of estimated person models in every successive set of frames, and tracks are built, several motion priors can be used, ranging from a constant velocity model, to more sophisticated auto-regressive processes or deep recursive predictors learned”
Part-Activated Deep Reinforcement Learning for Action Prediction (2018, background): “In spite of the enormous amount of works conducted in this area [21, 25, 26], this task is still challenging due to the fundamental challenges inherent in the problem like the large variance among the activities and huge spatiotemporal scale variation.”
Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control (2018, background): “This is a common problem in learning a model to reproduce a given distribution of trajectories (Ross et al., 2010; Bengio et al., 2015; Martinez et al., 2017; Lamb et al., 2016).”
Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, methodology): “The motion prediction helps machines understand human behaviors, attracting considerable attention [9, 20, 32, 5, 12, 2].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “Approaches mainly focus on efficiently and effectively fusing spatial and temporal information to generate deterministic future motion through different models: RNN [12,20,50,51], GAN [9,30], GCN [49], Attention [48] or even simply MLP [11, 17, 25].”
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison (2019, methodology): “Pose based approaches mainly utilize RNNs [45] to model the pose sequences for analyzing human motions.”
Human Motion Diffusion Model (2022, background): “Some predict motion from its preﬁx poses (Fragkiadaki et al., 2015; Martinez et al., 2017; Hernandez et al., 2019; Guo et al., 2022b).”
QuaterNet: A Quaternion-based Recurrent Model for Human Motion (2018, methodology): “Similar to [37], we found this approach to be beneficial for short-term prediction, but we also discovered that it leads to instability in the long-term.”
Learning Trajectory Dependencies for Human Motion Prediction (2019, methodology): “Because of their success at sequence-to-sequence prediction [21, 14], RNNs have become the de facto model for human motion prediction [7, 11, 17].”
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, methodology): “Most methods generate the motion of 3D skeletons [13, 22, 38–40, 61 ], while others [ Motion generation for bodies in 3D scenes: Most early methods extend MoCap databases with point annotations for foot and hand contact [12, 25, 30, 31].”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, methodology): “after some time [46] and are subject to drift.”
Deep Learning-Based Vehicle Behavior Prediction for Autonomous Driving Applications: A Review (2020, background): “Despite having less parameters and complexity, single layer LSTMs are reported to achieve competitive results compared to the multilayer counterpart in some tasks [56], [57].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “sample diversity and accuracy. 2 Related Work Human Motion Prediction. Most previous work takes a deterministic approach to modeling human motion and regress a single future motion from past 3D poses [17, 34, 9, 45, 18, 52, 57, 13, 22, 1, 69, 51] or video frames [10, 75, 74]. While these approaches ”
Convolutional Sequence to Sequence Model for Human Dynamics (2018, methodology): “[14] solved the problem by adding a residual unit in the recurrent network.”
HP-GAN: Probabilistic 3D Human Motion Prediction via GAN (2017, methodology): “Lastly, we used standard GRU in our sequence-tosequence network and we did not investigate new RNN architecture that might provide better human pose prediction such as RNN with skip connection [23] or Part-Aware LSTM [32] or something else, this is another area we plan to investigate in the future.”