Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

By Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, Michael J. Black. 2016

1,816 citations recorded by the source corpus.

We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, methodology): “Figure 9: Qualitative comparison between model-based single-person pose estimation of SMPLify [2] and our FCN-based result, in the absence (‘full-body images’) and presence (‘all images’) of occlusions.”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “Unlike previous approaches that make a priori assumptions about the joint limits [5, 52], we do not predefine the degrees of freedom of the kinematic skeleton model.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, methodology): “† We reﬁne the SPIN estimated pose using SMPLify.”
Benchmarking and Error Diagnosis in Multi-instance Pose Estimation (2017, background): “…for improvement are possible: (i) collecting 3D annotations [7] for the humans in COCO and learning to directly regress 3D pose end-to-end [30]; (ii) modeling the manifold of human poses [3, 6, 35] and learning how to jointly predict the 3D pose of a person along with its 2D skeleton [41].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Human reconstruction and performance capture Perceiving humans from monocular RGB data [12, 29, 31, 41, 43, 44, 58, 59, 64, 87] and under multiple views [37–40, 62] settings has been widely explored.”
3D Human Pose Estimation from a Single Image via Distance Matrix Regression (2016, methodology): “The results of all approaches are obtained from the original papers, except for (*), which are from [10].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “The iterative fitting [4,34], neural network [17], or their combination [21] can be used to obtain more data.”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “Other methods created over-complete bases which are suitable for representing human poses as sparse combinations [2, 4, 44, 62, 77].”
3D Human Pose Estimation in the Wild by Adversarial Learning (2018, background): “Significant progress has been achieved for 3D human pose estimation from monocular images due to the availability of large-scale dataset [2] and the powerful DCNNs.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Collision penalization is implemented either with approximate shape primitives [3, 34, 48] or triangle meshes [2, 16, 17, 19, 23, 33, 50, 60, 65].”
LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “Leveraging 3D templates (Roth et al., 2016; Goel et al., 2020; Kanazawa et al., 2018; Kulkarni et al., 2020), semantics (Li et al., 2020), and poses (Bogo et al., 2016; Novotny et al., 2019) as shape priors have also been widely studied in category-specific reconstruction.”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Some works split the problem in two: ﬁrst estimate 2D joints and then lift them to 3D [58, 54, 9, 66, 35, 69, 1, 52, 20, 32, 6, 26, 38, 57, 2], e.g ., by database matching, neural network regression, or ﬁtting the SMPL body model [30].”
Monocular, One-stage, Regression of Multiple 3D People (2020, methodology): “Impressive performance has been achieved using various weak supervision signals, such as 2D pose [9, 20, 25], semantic segmentation [51], geometric priors and discriminative training [20], motion dynamics [21], temporal coherence [22, 47], texture consistency [39], optimization [6] in the loop [23],”
Deep 3D human pose estimation: A review (2021, neutral): “As a well-known method, the SMPLify (Bogo et al., 2016) first estimates 2D keypoints using DeepCut (Pishchulin et al., 2016) and then fits the SMPL model to these keypoints.”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “Some methods fit the SMPL model to the detected 2D keypoints [3, 25].”
Local Deep Implicit Functions for 3D Shape (2019, methodology): “is dataset is quite different than for ShapeNet – the autoencoder must be able to represent large-scale, nonFigure 7. Human body modeling. Surface reconstructions and decompositions for 4 random SMPL [5] human meshes from the SURREAL [37] dataset. For each triple, from left to right: SMPL mesh, our ”