Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision

By Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, Christian Theobalt. 2016

1,341 citations recorded by the source corpus.

We propose a CNN-based approach for 3D human body pose estimation from single RGB images that addresses the issue of limited generalizability of models trained solely on the starkly limited publicly available 3D pose data. Using only the existing 3D pose data and 2D pose data, we show state-of-the-art performance on established benchmarks through transfer of learned features, while also generalizing to in-the-wild scenes. We further introduce a new training set for human body pose estimation from monocular images of real humans that has the ground truth captured with a multi-camera marker-less motion capture system. It complements existing corpora with greater diversity in pose, human appearance, clothing, occlusion, and viewpoints, and enables an increased scope of augmentation. We also contribute a new benchmark that covers outdoor and indoor scenes, and demonstrate that our 3D pose dataset shows better in-the-wild performance than existing annotated data, which is further improved in conjunction with transfer learning from 2D pose data. All in all, we argue that the use of transfer learning of representations in tandem with algorithmic and data contributions is crucial for general 3D body pose estimation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, methodology): “6M and MPII are used for training, while the testing is performed on MPI-INF-3DHP.”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “Still others [27, 28] use pre-trained 2D pose networks and also use 2D pose prediction as an auxiliary task.”
Compositional Human Pose Regression (2017, methodology): “While they are widely used for 3D pose estimation [54, 30, 28, 43, 24, 42, 33], the performance is not satisfactory.”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Other works proposed similar methods for monocular pose estimation [25, 18, 27].”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, background): “Some works transfer the features learned for 2D pose estimation to the 3D task [35].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, methodology): “Evaluation Metric : We use the robust 3DPCK evaluation metric proposed in [33].”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, neutral): “Results on MPI-INF-3DHP.”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Many existing works [3, 12, 17, 29, 54, 53, 15, 14] rely on effective 2D pose estimation frameworks to first localize the 2D keypoints on the image plane, and then lift 3D poses from the estimated 2D joint positions.”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “6M, MSCOCO, MPII [1], LSP [17], LSP-Extended [18], UP [25], and MPI-INF-3DHP [30].”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “to 2D joints using SMPLify [6]. Following them, we use the processed data for training. MuCo-3DHP. MuCo-3DHP [41] is synthesized from the existing MPI-INF3DHP 3D single-person pose estimation dataset [40]. It consists of 200K frames, and half of them have augmented backgrounds. For the background au”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Other works regress poses directly from pixels [46, 55, 56, 68, 71, 77, 78, 96], which generally relies on having a good match between training and testing.”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, positive): “6M [16] and MPI-INF-3DHP [35]), and 2D datasets (COCO [30] and MPII [2]) with pseudo-GT provided by the CLIFF annotator.”
Monocular, One-stage, Regression of Multiple 3D People (2020, background): “6M [14] and MPI-INF-3DHP [34]), one pseudo-label 3D dataset (UP [26]) and four in-the-wild 2D pose datasets (MS COCO [30], MPII [2], LSP [16, 17] and AICH [50]).”
Deep 3D human pose estimation: A review (2021, positive): “For this purpose, Mehta et al. (2017b) first estimate both 2D and root (pelvis) relative 3D pose using a CNN.”
VNect (2017, methodology): “For 3D pose, we use MPI-INF3DHP [Mehta et al. 2016] and Human3.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, background): “images withMoCap 3D poses and real-world images with 2D annotations [50], [51].”
On Self-Contact and Human Pose (2021, methodology): “We evaluate TUCH on the following three datasets: 3DPW [43], MPI-INF-3DHP [30], and 3DCP Scan.”
Learning to Estimate 3D Human Pose and Shape from a Single Color Image (2018, background): “Many recent works follow the end-to-end paradigm [48, 40, 42, 46, 55], using images as input to predict 3D joint locations [23, 45, 34, 28], regress 3D heatmaps [31], or classify the image in a particular pose class [39, 40].”