Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose

By Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, Kostas Daniilidis. 2016

983 citations recorded by the source corpus.

This paper addresses the challenge of 3D human pose estimation from a single color image. Despite the general success of the end-to-end learning paradigm, top performing approaches employ a two-step solution consisting of a Convolutional Network (ConvNet) for 2D joint localization and a subsequent optimization step to recover 3D pose. In this paper, we identify the representation of 3D pose as a critical issue with current ConvNet approaches and make two important contributions towards validating the value of end-to-end learning for this task. First, we propose a fine discretization of the 3D space around the subject and train a ConvNet to predict per voxel likelihoods for each joint. This creates a natural representation for 3D pose and greatly improves performance over the direct regression of joint coordinates. Second, to further improve upon initial estimates, we employ a coarse-to-fine prediction scheme. This step addresses the large dimensionality increase and enables iterative refinement and repeated processing of the image features. The proposed approach outperforms all state-of-the-art methods on standard benchmarks achieving a relative error reduction greater than 30% on average. Additionally, we investigate using our volumetric representation in a related architecture which is suboptimal compared to our end-to-end approach, but is of practical interest, since it enables training when no image with corresponding 3D groundtruth is available, and allows us to present compelling results for in-the-wild images.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, background): “Recently, they have also been successfully applied to 3D pose estimation [19, 25, 30, 32, 39, 42, 44].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “3D pose estimation tasks thanks to the rise of new architectures and the availability of large amounts of data [33, 35].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “[45], many recent methods estimate 3D joints directly from images in a deep learning framework [33, 43, 44, 51, 52].”
Compositional Human Pose Regression (2017, methodology): “Unlike the above methods, our approach treats the 2D and 3D data in the same way and combine them in a unified training framework.”
BodyNet: Volumetric Inference of 3D Human Body Shapes (2018, methodology): “Extending the notion of 2D heatmaps to 3D, we represent 3D joint locations with 3D Gaussians deﬁned on a voxel grid as in [6].”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, methodology): “[19] provided an alternative decoupled version which can also be applied in the wild, but its MPJPE increased to 78.”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, neutral): “[39] and Moon et al.”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, methodology): “Walking and upright poses are prime examples of this, and state-of-the-art algorithms [18, 27, 20, 15, 17] now deliver impressive real-time results in uncontrolled environments.”
Learning to Estimate 3D Hand Pose from Single RGB Images (2017, background): “[15] proposed a volumetric approach that treats pose estimation as per voxel prediction of scores in a coarse-to-fine manner, which gives a natural representation to the data, but is computationally expensive and limited by the GPU memory to fit the voxel grid.”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “The closest work to ours are the approaches of [29, 30, 56, 58] in that they also perform 2.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Other works regress poses directly from pixels [46, 55, 56, 68, 71, 77, 78, 96], which generally relies on having a good match between training and testing.”
Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression (2017, methodology): “Different from [16], we demonstrate that a 3D volumetric representation is particular effective for learning dense 3D facial geometry.”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “Compared to [41, 40], we do not use heatmaps and instead describe poses with detected keypoint coordinates.”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, background): “The research can be categorized into sparse 2D parsing (2D skeleton estimation) [70, 50, 5, 47], sparse 3D parsing (3D skeleton estimation) [41, 49, 53, 78, 59, 62, 44, 74], dense 2D parsing [18, 17] and dense 3D parsing (shape and pose estimation).”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, methodology): “It is common protocols [14, 15, 6] to use five subjects (S1, S5, S6, S7, S8) for training and two subjects (S9, S11) for evaluation.”
Benchmarking and Error Diagnosis in Multi-instance Pose Estimation (2017, background): “…for improvement are possible: (i) collecting 3D annotations [7] for the humans in COCO and learning to directly regress 3D pose end-to-end [30]; (ii) modeling the manifold of human poses [3, 6, 35] and learning how to jointly predict the 3D pose of a person along with its 2D skeleton [41].”
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints (2018, methodology): “Our work relates to recently developed deep architectures for 2d human pose estimation [4, 9, 21, 35, 36], 3d human pose estimation based on fitting volumetric models [2, 15], feedforward deep models for 3d prediction [18, 22, 40], as well as integrated deep models for 2d and 3d reasoning [23, 27, 3”
Human Pose Regression with Residual Log-likelihood Estimation (2021, methodology): “Existing methods can be divided into two categories: heatmap-based [57, 56, 62, 4, 64, 54, 46, 52] and regression-based [58, 5, 53, 70, 42, 61].”