Learning Trajectory Dependencies for Human Motion Prediction

By Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li. IEEE International Conference on Computer Vision, 2019

559 citations recorded by the source corpus.

Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/bad2b0aa0a001de221ca86838372acfb99e68e49

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, methodology): “We see that DMGNN achieves the shortest running time while generating future poses over both 400 or 1000 ms, compared with the other competitors [32, 26, 31].”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “, 2020), future human motions (Fragkiadaki et al., 2015; Mao et al., 2019), etc.”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, background): “Early work on motion generation adopts deterministic human motion modeling which only generates a single motion [2, 17, 18, 20, 39, 51, 52, 57, 85, 90].”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, background): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”
History Repeats Itself: Human Motion Prediction via Motion Attention (2020, methodology): “6 LTD-50-25[20] 12.”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “Approaches mainly focus on efficiently and effectively fusing spatial and temporal information to generate deterministic future motion through different models: RNN [12,20,50,51], GAN [9,30], GCN [49], Attention [48] or even simply MLP [11, 17, 25].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “sample diversity and accuracy. 2 Related Work Human Motion Prediction. Most previous work takes a deterministic approach to modeling human motion and regress a single future motion from past 3D poses [17,34,9,45,18,52,57,13,22,1,69,51] or video frames [10,77,75]. While these approaches are able to p”