MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior

By Xiaowei Zhou, Menglong Zhu, Georgios Pavlakos, Spyridon Leonardos, Konstantinos G. Derpanis, Kostas Daniilidis. 2017

225 citations recorded by the source corpus.

Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D, and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to “in-the-wild” images, which is demonstrated with the MPII dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, methodology): “In testing, the estimated 3D pose is re-scaled to the size of a pre-defined canonical skeleton, as done in [10], [40].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “Recently, deep architectures have been used to learn precise 3D representations from RGB images [60, 50, 30, 49, 31, 39], thanks to the availability of high quality data [21], and are now able to surpass depthsensors [32].”
Integral Human Pose Regression (2017, methodology): “Some works [51,41,8,4,32,54] ﬁrstly align the predicted 3D pose and ground truth 3D pose with a rigid transformation using Procrustes Analysis [18] and then compute MPJPE.”
Weakly-supervised Transfer for 3D Human Pose Estimation in the Wild (2017, result): “Note that this is already comparable to most state-of-theart methods [33, 26, 35], indicating the effectiveness of our network architecture.”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, unknown): “[60], is another metric for this task.”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, background): “This is already comparable to most state-of-the-art methods [33, 26, 35].”
Deep Learning-based Human Pose Estimation: A Survey (2020, unknown): “, [135] [136] [137] [138] [139] [140] [141] [142].”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “[68, 69] use 2D heatmaps from a 2D pose ConvNet to reconstruct 3D pose in a video sequence.”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, methodology): “We present results for two popular protocols (P1 and P2, as defined in [15]) and two error metrics (MPJPE and Reconstruction error, as defined in [51]).”
Self-supervised Learning of Motion Capture (2017, result): “Evaluation metrics Given predicted 3D body joint locations of K = 32 keypoints Xkpt, k = 1 · · ·K and corresponding ground-truth 3D joint locations X̃kpt, k = 1 · · ·K, we define the per-joint error of each example as 1 K ∑K k=1 ‖Xkpt − X̃kpt‖2 similar to previous works [41].”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “we require that the sum of length of all 3D bones is equal to that of a canonical skeleton as shown in [41, 75, 78].”
Mesh Graphormer (2021, methodology): “We evaluate our model using the 3D pose accuracy, and report the accuracy using PA-MPJPE metric [43, 14, 19, 20].”
Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision (2017, methodology): “We show the 3D reconstruction error (in millimeters) of our model and baselines in Tables 1 and 2, organized according to activity, following the presentation format of Zhou et al. [39], though only one model was used across all activities for our method and baselines (for MonoCap this means using t”