Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

By Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll. 2020

220 citations recorded by the source corpus.

Implicit functions represented as deep learning approximations are powerful for reconstructing 3D surfaces. However, they can only produce static surfaces that are not controllable, which provides limited ability to modify the resulting model by editing its pose or shape parameters. Nevertheless, such features are essential in building flexible models for both computer graphics and computer vision. In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing. Given sparse 3D point clouds sampled on the surface of a dressed person, we use an Implicit Part Network (IP-Net)to jointly predict the outer 3D surface of the dressed person, the and inner body surface, and the semantic correspondences to a parametric body model. We subsequently use correspondences to fit the body model to our inner surface and then non-rigidly deform it (under a parametric body + displacement model) to the outer surface in order to capture garment, face and hair detail. In quantitative and qualitative experiments with both full body data and hand scans we show that the proposed methodology generalizes, and is effective even given incomplete point clouds collected from single-view depth images. Our models and code can be downloaded from this http URL.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, methodology): “This shows that our formulation is superior than IPNet even for human registration.”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “Given a sparse point cloud of a clothed person, IPNet [10] infers an occupancy ﬁeld with body/clothing layers, registers SMPL to the body layer with inferred body-part segmentation, and captures clothing as offsets from SMPL to the point cloud.”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, methodology): “[18, 5] combine implicit function learning with the SMPL model to obtain detailed animatable human models.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, background): “There have been a growing number of papers on implicit function learning (IFL) for shape representation and reconstruction tasks [53, 14, 49, 68, 15, 27, 11, 10].”
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors (2021, methodology): “Finally, by explicitly encoding depth observations using the truncated PSDF values, the proposed GeoNet can not only achieves accurate reconstruction results but also orders of magnitude faster than IPNet (approximately 80 seconds for reconstruction).”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “The success of deep implicit representations in general object modeling has inspired research in 3D human reconstruction [8], [54], [55], [56], [57].”