Exploiting Temporal Information for 3D Human Pose Estimation

By Mir Rayat Imtiaz Hossain, James J. Little. European Conference on Computer Vision, 2017

367 citations recorded by the source corpus.

In this work, we address the problem of 3D human pose estimation from a sequence of 2D human poses. Although the recent success of deep networks has led many state-of-the-art methods for 3D pose estimation to train deep networks end-to-end to predict from images directly, the top-performing approaches have shown the effectiveness of dividing the task of 3D pose estimation into two steps: using a state-of-the-art 2D pose estimator to estimate the 2D pose from images and then mapping them into 3D space. They also showed that a low-dimensional representation like 2D locations of a set of joints can be discriminative enough to estimate 3D pose with high accuracy. However, estimation of 3D pose for individual frames leads to temporally incoherent estimates due to independent error in each frame causing jitter. Therefore, in this work we utilize the temporal information across a sequence of 2D joint locations to estimate a sequence of 3D poses. We designed a sequence-to-sequence network composed of layer-normalized LSTM units with shortcut connections connecting the input to the output on the decoder side and imposed temporal smoothness constraint during training. We found that the knowledge of temporal consistency improves the best reported result on Human3.6M dataset by approximately \(12.2\%\) and helps our network to recover temporally consistent 3D poses over a sequence of images even when the 2D pose detector fails.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/38cc89399dd6f5aaab1654f27ab3c9eeade12a36

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation (2021, methodology): “This protocol is referred to as protocol #2 [19], [23].”
Integral Human Pose Regression (2017, methodology): “While regression methods are widely used for 3D pose estimation [42,55,56,31,32,30,35,43,21,14], its performance is still not satisfactory.”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “Following [34, 16], we evaluate on three actions (Walk, Jog, Box) by training a different model for each action (single action – SA).”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, background): “Remaining works target at exploiting temporal information [11, 18, 21, 57] for 3D pose regression.”
Recent Advances in Deep Learning for Object Detection (2019, background): “FPN achieved significant progress in detecting multi-scale objects and has been widely used in many other domains such as video detection [53, 54] and human pose recognition [55, 56].”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, methodology): “In this research, we follow the evaluation protocols in prior work [17, 21, 26, 28, 37, 39], in which 5 subjects (S1, S5, S6, S7, S8) are used for training and 2 subjects (S9 and S11) are adopted for testing.”
Learnable Triangulation of Human Pose (2019, neutral): “8 Hossain & Little [2] (∗) 48.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “Hossain and Little [202] proposed a recurrent neural network using a Long Short-Term Memory (LSTM) unit with shortcut connections to exploit temporal information from sequences of human pose.”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, methodology): “Then the temporal consistency loss (TCLoss) following [12] is employed to improve the temporal smoothness performance (MPJVE) by 1.”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, background): “Hossain and Little [33] proposed a recurrent neural network using Long Short-Term Memory (LSTM) cells to exploit temporal information in the input sequence.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “Several methods [14, 24, 52] use a two-stage approach to “lift” off-the-shelf 2D keypoint detection results into 3D joint locations.”
Learning 3D Human Dynamics From Video (2018, background): “Several approaches train LSTM models on various inputs such as image features [34], 2D joints [25], or 3D joints [12] to obtain temporally coherent 3D joint outputs.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “In case of video inputs, temporal constraints have been used for body motion estimation in the context of neural networks [20, 24], or optimization [1, 51].”