3D Human Pose Estimation from a Single Image via Distance Matrix Regression

By Francesc Moreno-Noguer. 2016

411 citations recorded by the source corpus.

This paper addresses the problem of 3D human pose estimation from a single image. We follow a standard two-step pipeline by first detecting the 2D position of the N body joints, and then using these observations to infer 3D pose. For the first step, we use a recent CNN-based detector. For the second step, most existing approaches perform 2N-to-3N regression of the Cartesian joint coordinates. We show that more precise pose estimates can be obtained by representing both the 2D and 3D human poses using NxN distance matrices, and formulating the problem as a 2D-to-3D distance matrix regression. For learning such a regressor we leverage on simple Neural Network architectures, which by construction, enforce positivity and symmetry of the predicted matrices. The approach has also the advantage to naturally handle missing observations and allowing to hypothesize the position of non-observed joints. Quantitative results on Humaneva and Human3.6M datasets demonstrate consistent performance gains over state-of-the-art. Qualitative evaluation on the images in-the-wild of the LSP dataset, using the regressor learned on Human3.6M, reveals very promising generalization results.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, methodology): “Two stage methods first predict 2D joint locations using 2D pose detectors [30, 49, 54] or ground truth 2D pose and then predict 3D joint locations from the 2D joints either by regression [26, 29] or model fitting, where a common approach exploits a learned dictionary of 3D skeletons [2, 34, 47, 37,”
Compositional Human Pose Regression (2017, methodology): “2D joints are first estimated and then used to estimate the 3D pose via learning a 2D to 3D mapping (Moreno-Noguer, 2016; Martinez et al., 2017; Fang et al., 2018), fitting a 3D model via optimization (Zhou et al.”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “A couple of algorithms directly predicted 3D pose from the image [75], while others combined 2D heatmaps with volumetric representation [41], pairwise distance matrix estimation [36] or image cues [56] for 3D human pose regression.”
Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation (2018, methodology): “Similar to model-based methods, learning approaches have benefited from the advent of robust 2D pose methods – by matching 2D detections to a 3D pose database [8, 66], by regressing pose from 2D joint distance matrices [35], by exploiting pose and geometric priors for lifting [69, 1, 51, 19, 32, 70,”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, methodology): “One straightforward approach is to ‘lift’ 2D poses into 3D, using either dictionaries or direct regression [3, 44, 47, 64, 79, 85, 88, 93].”
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “Additionally, there are two-stage approaches which de-compose the 3D pose inference task into two independent stages: estimating 2D poses, and lifting them into 3D space [8, 24, 22, 11, 46, 8, 40, 23].”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, background): “More recent approaches, on the other hand, use deep neural networks to learn a mapping from 2D pose to 3D [44–46].”
Integral Human Pose Regression (2017, methodology): “Regression methods are widely used for 3D pose estimation [38, 45, 46, 25, 27, 24, 30, 39].”
3D Human Pose Estimation in the Wild by Adversarial Learning (2018, background): “In some works, the predictions are further aligned with the ground-truth via a rigid transform [2, 28, 25], which is referred as Protocol #2.”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Two-stage approaches first estimate 2D pose and then lift them to 3D joint locations by a learned dictionary of 3D skeleton [1, 53, 64, 56, 76, 77] or regression [46, 71, 40, 11, 32, 59].”
Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image (2019, neutral): “e Yasin [50] 88.4 72.5 108.5 110.2 97.1 81.6 107.2 119.0 170.8 108.2 142.5 86.9 92.1 165.7 102.0 108.3 Chen [5] 71.6 66.6 74.7 79.1 70.1 67.6 89.3 90.7 195.6 83.5 93.3 71.2 55.7 85.9 62.5 82.7 Moreno [32] 67.4 63.8 87.2 73.9 71.5 69.9 65.1 71.7 98.6 81.3 93.3 74.6 76.5 77.7 74.6 76.5 Zhou [53] 47.9 ”
Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision (2016, background): “Recent advances in direct CNNbased 3D regression show promise, utilizing different prediction space formulations [60, 31, 76, 40, 36] and incorporating additional constraints [76, 62, 78, 73].”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “The methods from [31], [33], [48], [51] or [67] report better performance for the remaining 7 actions.”
C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “Besides the fully supervised methods [25, 26], several works have explored multi-view supervision [20, 29, 31], ordinal depth supervision [28], unpaired 2D-3D data [30, 36, 41, 15] or videos [17] to alleviate the need for full 2D-3D annotations.”
Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation (2017, methodology): “We compare our method with 6 state-ofthe-art methods (Simo-Serra et al. 2013; Kostrikov and Gall 2014; Yasin et al. 2016; Moreno-Noguer 2017; Pavlakos et al. 2017; Martinez et al. 2017).”
Learning to Estimate 3D Hand Pose from Single RGB Images (2017, background): “Recently, there have been several approaches that apply deep learning for lifting 2D keypoints to 3D pose for human body pose estimation [26, 11, 16].”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, background): “Alternatively, a network can also handle the step of lifting 2D estimates to 3D poses [56, 28, 51].”