Unite the People: Closing the Loop Between 3D and 2D Human Representations

By Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J. Black, Peter Gehler. 2017

585 citations recorded by the source corpus.

3D models provide a common ground for different representations of human bodies. In turn, robust 2D estimation has proven to be a powerful tool to obtain 3D fits in-the-wild. However, depending on the level of detail, it can be hard to impossible to acquire labeled data for training 2D estimators on large scale. We propose a hybrid approach to this problem: with an extended version of the recently introduced SMPLify method, we obtain high quality 3D body model fits for multiple human pose datasets. Human annotators solely sort good and bad fits. This procedure leads to an initial dataset, UP-3D, with rich annotations. With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body. Using the 91 landmark pose estimator, we present state-of-the art results for 3D human pose and shape estimation using an order of magnitude less training data and without assumptions about gender or pose in the fitting procedure. We show that UP-3D can be enhanced with these improved fits to grow in quantity and quality, which makes the system deployable on large scale. The data, code and models are available for research purposes.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, methodology): “We use the code provided by [23] with both DeeperCut pose estimation landmark detector [18] for 14-landmark results and with the 91landmark alternative proposed in [23].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “We report the segmentation accuracy and average F1 score over all parts including the background as done in [20].”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, background): “Methods that involve part segmentation as input [33, 42] can produce more accurate fittings.”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “This optimization paradigm was further extended with silhouette cues [27], volumetric grid [67], multiple people [75] and whole-body parametric model [49].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “The most straightforward way is by fitting a 3D body model [48] to 2D pose detections [10, 43].”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “For example, by ﬁtting the SMPL model to the 2D keypoint detections [34] and other dense shape cues [35], the shape and pose parameters can be automatically obtained from a single image [10], [36].”
3D Menagerie: Modeling the 3D Shape and Pose of Animals (2016, background): “ kp() = X k ˆ(jjx 1 jk mj jXk m j=1 ( v k j ;) jj 2); (12) where ˆis the Geman-McClure robust error function [15]. Silhouette reprojection. We encourage silhouette coverage and consistency similar to [19, 21, 33] using a bi-directional distance: E silh() = X x2S^ D S(x) + X x2S ˆ(min ^x2S^ jjx ^xjj ”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “[56] proposed a selfimproving, scalable method that obtains high-quality 3D body model fits for 2D images.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “[49] Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”
Mesh Graphormer (2021, methodology): “6M [12], MuCo-3DHP [25], UP-3D [21], COCO [23], MPII [1].”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, methodology): “Several methods estimate the SMPL model from a single image [37, 41, 59, 62].”
Learning the Depths of Moving People by Watching Frozen People (2019, background): “Recovery of a posed 3D human mesh from a single RGB image has attracted signiﬁcant attention [18, 9, 16, 1, 28, 23].”
Deep 3D human pose estimation: A review (2021, unknown): “The SMPL model based methods (Arnab et al., 2019a; Liang and Lin, 2019; Huang et al., 2017; Bogo et al., 2016; Kanazawa et al., 2018; Lassner et al., 2017) are comparable to other methods.”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, background): “[21] estimate proposals of 3D human shape fits for single images which are accepted or rejected by annotators.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “(2020) for a broader review, it is worth mentioning that related work includes optimization-based approaches, which fit a 3D mesh to 2D image keypoints (Bogo et al., 2016; Kolotouros et al., 2019a; Lassner et al., 2017; Yang and Carlone, 2020; Zanfir et al., 2018), and learning-based methods, which ”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “[39] uses silhouettes along with keypoints for the fitting algorithm.”
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020, methodology): “To overcome such ambiguity, parametric 3D models [5, 27, 18, 33] are often used to restrict estimation to a small set of model parameters, constraining the solution space to a speciﬁcally chosen parametric body model [7, 22, 20, 46, 33, 47].”
Learning to Dress 3D People in Generative Clothing (2019, unknown): “Most approaches [9, 17, 23, 26, 27, 30, 36, 38, 46] output 3D body meshes from images, but not clothing.”