Learning to Estimate 3D Human Pose and Shape from a Single Color Image

By Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, Kostas Daniilidis. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

647 citations recorded by the source corpus.

This work addresses the problem of estimating the full body 3D human pose and shape from a single color image. This is a task where iterative optimization-based solutions have typically prevailed, while Convolutional Networks (ConvNets) have suffered because of the lack of training data and their low resolution 3D predictions. Our work aims to bridge this gap and proposes an efficient and effective direct prediction method based on ConvNets. Central part to our approach is the incorporation of a parametric statistical body shape model (SMPL) within our end-to-end framework. This allows us to get very detailed 3D mesh results, while requiring estimation only of a small number of parameters, making it friendly for direct network prediction. Interestingly, we demonstrate that these parameters can be predicted reliably only from 2D keypoints and masks. These are typical outputs of generic 2D human analysis ConvNets, allowing us to relax the massive requirement that images with 3D shape ground truth are available for training. Simultaneously, by maintaining differentiability, at training time we generate the 3D mesh from the estimated parameters and optimize explicitly for the surface using a 3D per-vertex loss. Finally, a differentiable renderer is employed to project the 3D mesh to the image, which enables further refinement of the network, by optimizing for the consistency of the projection with 2D annotations (i.e., 2D keypoints or masks). The proposed approach outperforms previous baselines on this task and offers an attractive solution for direct prediction of3D shape from a single color image.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/13ad6164dba75845f0f397e9314ad596e74eb946

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “ining. However, models trained on the synthetic dataset usually produce unsatisfactory estimation results on real-world datasets due to the domain gap between them. To address this issue, inspired by [4, 37], we propose a novel weaklysupervised method by leveraging depth map as a weak supervision fo”
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera (2018, background): “Some approaches directly predict the parameters of a body model (SMPL) from a single image using 2D supervision [10, 22, 40].”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, neutral): “53, 107, 109], and learning-based methods, which infer the mesh directly from pixel information [38, 44, 45, 77, 79, 96].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, background): “Various types of inputs are considered in these deep network approaches, including single RGB images [20], 2D keypoint heatmaps [39], body part segmentation [35] or densepose maps [44, 56].”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “Since it is challenging to regress the pose and shape coefficients directly from an input image, recent works further propose to leverage various human body priors such as human skeletons [21, 34] or segmentation maps [29], and explore different optimization strategies [19, 17, 46, 12] and temporal ”
NASA: Neural Articulated Shape Approximation (2019, methodology): “Although parametric models of human body such as SMPL [33] have been integrated into neural network frameworks for self-supervision [27,38,40,55], these approaches depend heavily on polygonal mesh representations.”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Other recent works integrate the SMPL model, or a voxel representation [74], as a layer within a network architecture [41, 55, 53, 73].”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, neutral): “On the other hand, regression-based methods [11, 25, 26] rely on deep neural networks and large amounts of training data [8, 23] to directly predict the 3D pose of the human.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “[18, 33] ﬁt such parametric models to CNN-regressed 2D joint positions to estimate hand poses from full-body images.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “…et al. 2017; Zanfir et al. 2018; Kolotouros et al. 2019; Yang and Carlone 2020), and learning-based methods, which infer the mesh directly from pixel information (Tan et al. 2017; Kanazawa et al. 2018; Omran et al. 2018; Pavlakos et al. 2018; Kolotouros et al. 2019b; Kolotouros et al. 2019).”
Learning to Dress 3D People in Generative Clothing (2019, background): “Deep learning methods reconstruct human shape from images, based on minimally dressed human models [5, 23, 26, 27, 30, 36, 38, 39].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, methodology): “Several methods estimate the SMPL model from a single image [37, 41, 59, 62].”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “[71] Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “Another line of work develops a learning-based framework, using a feed-forward model to directly predict the parameters of the body model from a single image [61,53,30,47,63,44].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “[40] used 2D joint heatmaps and silhouette as cues for predicting accurate SMPL parameters.”