Deep Representation Learning for Human Motion Prediction and Classification

By Judith Bütepage, Michael J. Black, Danica Kragic, Hedvig Kjellström. 2017

443 citations recorded by the source corpus.

Generative models of 3D human motion are often restricted to a small number of activities and can therefore not generalize well to novel movements or applications. In this work we propose a deep learning framework for human motion capture data that learns a generic representation from a large corpus of motion capture data and generalizes well to new, unseen, motions. Using an encoding-decoding network that learns to predict future 3D poses from the most recent past, we extract a feature representation of human motion. Most work on deep learning for sequence prediction focuses on video and speech. Since skeletal data has a different structure, we present and evaluate different network architectures that make different assumptions about time dependencies and limb correlations. To quantify the learned features, we use the output of different layers for action classification and visualize the receptive fields of the network units. Our method outperforms the recent state of the art in skeletal motion prediction even though these use action specific training data. Our results show that deep feedforward networks, trained from a generic mocap database, can successfully be used for feature extraction from human motion data and that this representation can be used as a foundation for classification and prediction.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

An Incremental Self-Organizing Architecture for Sensorimotor Learning and Prediction (2017, neutral): “accurate than recurrent approaches [50].”
Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, background): “The motion prediction helps machines understand human behaviors, attracting considerable attention [9, 20, 32, 5, 12, 2].”
History Repeats Itself: Human Motion Prediction via Motion Attention (2020, background): “In view of the drawbacks of RNNs, several works considered feed-forward networks as an alternative solution [4,17,20].”
Convolutional Sequence to Sequence Model for Human Dynamics (2018, methodology): “The main task of this work is human motion prediction via convolutional models, while previous works[3, 5, 14] mainly focus on RNN based models.”
A Spatio-temporal Transformer for 3D Human Motion Prediction (2020, methodology): “However, considering the skeletal structure in the architectural level is shown to be an effective inductive bias in [2, 4, 20, 24].”
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
Learning Trajectory Dependencies for Human Motion Prediction (2019, methodology): “In [5], this was achieved by exploiting the human skeleton, and in [16] by defining a relatively large spatial filter size.”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “Most previous work takes a deterministic approach to modeling human motion and regress a single future motion from past 3D poses [17, 34, 9, 45, 18, 52, 57, 13, 22, 1, 69, 51] or video frames [10, 77, 75].”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, methodology): “Other methods use past 3D human pose sequence as input to predict future human motions [11, 16, 5, 26].”
Learning 3D Human Dynamics From Video (2018, unknown): “Figure 2 illustrates the framework.”