Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video

By Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Konstantinos G. Derpanis, Kostas Daniilidis. 2015

441 citations recorded by the source corpus.

This paper addresses the challenge of 3D full-body human pose estimation from a monocular image sequence. Here, two cases are considered: (i) the image locations of the human joints are provided and (ii) the image locations of joints are unknown. In the former case, a novel approach is introduced that integrates a sparsity-driven 3D geometric prior and temporal smoothness. In the latter case, the former case is extended by treating the image locations of the joints as latent variables to take into account considerable uncertainties in 2D joint locations. A deep fully convolutional network is trained to predict the uncertainty maps of the 2D joint locations. The 3D pose estimates are realized via an Expectation-Maximization algorithm over the entire sequence, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Empirical evaluation on the Human3.6M dataset shows that the proposed approaches achieve greater 3D pose estimation accuracy over state-of-the-art baselines. Further, the proposed approach outperforms a publicly available 2D pose estimation baseline on the challenging PennAction dataset.

Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

In Human-Computer Interaction

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, methodology): “Two stage methods first predict 2D joint locations using 2D pose detectors [30, 49, 54] or ground truth 2D pose and then predict 3D joint locations from the 2D joints either by regression [26, 29] or model fitting, where a common approach exploits a learned dictionary of 3D skeletons [2, 34, 47, 37,”
LCR-Net: Localization-Classiﬁcation-Regression for Human Pose (unknown): “We evaluate only on every 5 th frame as in [37] , i.e ., on a test set of 110,000 images, as we did not observe a signiﬁcant impact on performance when evaluating on all the frames.”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “[68, 69] use 2D heatmaps from a 2D pose ConvNet to reconstruct 3D pose in a video sequence.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “We evaluate only on every 5th frame as in [44], i.”
6-DoF object pose from semantic keypoints (2017, background): “, [22], [23], [24], [25]) has attracted considerable study, while limited attention has been given to their application with generic object categories [26], [20].”
Deep Kinematic Pose Regression (2016, methodology): “The points could further be the physical joints that deﬁnes the geometry of complex articulated objects, such as human hand [41,21] and human body [17,40,31].”
Monocular 3D Human Pose Estimation by Predicting Depth on Joints (2017, methodology): “We also report results for protocol 2 (P2) which is employed in [41, 30, 23].”
A Simple Yet Effective Baseline for 3d Human Pose Estimation (2017, methodology): “2d/3d positions Our first design choice is to use 2d and 3d points as inputs and outputs, in contrast to recent work that has used raw images [11,13,24,32,33,45,46,54,56] or 2d probability distributions [33, 56] as inputs, and 3d probabilities [33], 3d motion parameters [54] or basis pose coefficien”
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints (2018, methodology): “Our work relates to recently developed deep architectures for 2d human pose estimation [4, 9, 21, 35, 36], 3d human pose estimation based on fitting volumetric models [2, 15], feedforward deep models for 3d prediction [18, 22, 40], as well as integrated deep models for 2d and 3d reasoning [23, 27, 3”
Exploiting Temporal Information for 3D Human Pose Estimation (2017, background): “Another common approach to estimating 3D joint locations given the 2D pose is to separate the camera pose variability from the intrinsic deformation of the human body, the latter of which is modeled by learning an over-complete dictionary of basis 3D poses from a large database of motion capture dat”
Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image (2017, background): “With the remarkable success in Structure from Motion (SfM), which is now capable of reconstructing entire cities using large-scale photo collections [1] and real-time visual SLAM on embedded and mobile devices [14], the computer vision community is starting to explore the possibility of constructing”
Deep Multitask Architecture for Integrated 2D and 3D Human Sensing (2017, methodology): “More recently, deep convolutional architectures have been employed in order to estimate 3d pose directly from images[21, 22, 34, 51] mostly in connection with 3d human motion capture datasets like HumanEva, Human3.”
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior (2017, background): “A preliminary version of this work appeared in CVPR 2016 [67].”
MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild (2016, methodology): “We also report results for a second protocol (P2) employed in [20, 44, 35] where all the frames from subjects S9 and S11 are used for testing and only S1, S5, S6, S7 and S8 are used for training.”
3D Human Pose Estimation from a Single Image via Distance Matrix Regression (2016, methodology): “This is the same evaluation protocol used by the baselines we compare against [3, 6, 8, 10, 12, 21, 30, 31, 38, 39, 40, 45, 47, 50, 52, 55, 56].”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Two-stage approaches ﬁrst estimate 2D pose and then lift them to 3D joint locations by a learned dictionary of 3D skeleton [1, 55, 66, 58, 78, 79] or regression [48, 73, 42, 11, 34, 61].”
Weakly-supervised Transfer for 3D Human Pose Estimation in the Wild (2017, methodology): “Following the standard protocol in [14, 34, 33], we use 5 subjects(S1, S5, S6, S7, S8) for training and the rest 2 subjects(S9, S11) for testing.”
Fusing 2D Uncertainty and 3D Cues for Monocular Body Pose Estimation (2016, methodology): “For completeness, we also compare our approach to the following methods that rely on either multiple consecutive images or impose temporal consistency: regression from short image sequences to 3D poses [59], fitting a sparse 3D pose model to 2D heatmap predictions across frames [69], and fitting a 3”