3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training

By Dario Pavllo, Christoph Feichtenhofer, David Grangier, Michael Auli. 2018

1,273 citations recorded by the source corpus.

In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce. Code and models are available at https://github.com/facebookresearch/VideoPose3D

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “In computer vision, representations for ImageNet (Deng et al., 2009) and COCO (Lin et al., 2014) have proven to be useful to initialize models for tasks such as image captioning (Vinyals et al., 2016) or pose estimation (Pavllo et al., 2019).”
Deep 3D human pose estimation: A review (2021, positive): “…also note that extracting spatio-temporal features using 3D CNNs directly on the volume does not improve the performance much compared to using spatial CNNs. Recently, Pavllo et al. (2019) propose a temporal dilated convolutional model taking 2D keypoint sequences as input to estimate 3D poses.”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, methodology): “We merge the TCLoss and MPJVE as the temporal loss function (T-Loss).”
Learnable Triangulation of Human Pose (2019, methodology): “At the same time, in such a regime, modern multi-view meth-ods have accuracy comparable to well-developed monocular methods [18, 12, 6, 16, 13].”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Instead of estimating 3D human pose from monocular images, videos can provide temporal information to improve accuracy and robustness of 3D HPE, e.g., References [13, 34, 44, 187, 227, 248, 311, 312].”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “Existing studies have shown the effectiveness of using 2D skeleton sequences for different downstream tasks [24,30,83,102].”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, methodology): “Following previous works [32, 25, 5], we adopt the same experiment setting: all 15 actions are used for training and testing, the model is trained on five sections (S1, S5, S6, S7, S8) and tested on two subjects (S9 and S11).”
Learning 3D Human Dynamics From Video (2018, methodology): “Concurrently to ours, [41] use a fully convolutional network to predict 3D joints from 2D joint sequences.”
Speech Recognition (2018, positive): “In computer vision, representations for ImageNet (Deng et al., 2009) and COCO (Lin et al., 2014) have proven to be useful to initialize models for tasks such as image captioning (Vinyals et al., 2016) or pose estimation (Pavllo et al., 2019).”
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, methodology): “Following previous works [4, 24, 34, 38], we train a single model on five subjects (S1, S5, S6, S7, S8) and test it on two subjects (S9 and S11).”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, methodology): “a large-scale dataset with 11 subjects (7 with ground truth) and 3.6 million video frames in total. Each subject performs 15 actions and the human motion is recorded at 50 Hz. Following previous work [53,49,73,56], we adopt a 17-joint skeleton and train on ve subjects (S1, S5, S6, S7, S8) and test o”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, methodology): “Note that [39] reported better results for 3D pose estimation using 243 frames.”
Human Pose Regression with Residual Log-likelihood Estimation (2021, methodology): “Two-stage methods [32, 13, 47, 68, 60, 9, 30, 67] lift the 2D poses to 3D space by regression.”
Speech gesture generation from the trimodal context of text, audio, and speaker identity (2020, methodology): “We further converted all human poses to 3D by using the 3D pose estimator (Pavllo et al. 2019) which convert a sequence of 2D poses into 3D poses. pose estimator uses temporal convolutions that lead to temporally coherent results despite of a few of inaccurate 2D poses.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, positive): “Furthermore, collecting 3D annotations is not as easy as in 2D. VideoPose3D [34] regresses 3D skeleton joint positions, by back-projecting them on the image space and using CNN-estimated 2D keypoints as supervision.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “Several methods [14, 24, 52] use a two-stage approach to “lift” off-the-shelf 2D keypoint detection results into 3D joint locations.”
Revisiting Skeleton-based Action Recognition (2021, methodology): “Besides RGB-based 3D-pose estimators, we also consider the ‘lifting’ approaches [42, 45], which directly ‘lift’ 2D-pose (sequences) to 3D-pose (sequences).”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, methodology): “[73] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.”