PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

By Shunsuke Saito, Tomas Simon, Jason M. Saragih, Hanbyul Joo. 2020

874 citations recorded by the source corpus.

Recent advances in image-based 3D human shape estimation have been driven by the significant improvement in representation power afforded by deep neural networks. Although current approaches have demonstrated the potential in real world settings, they still fail to produce reconstructions with the level of detail often present in the input images. We argue that this limitation stems primarily form two conflicting requirements; accurate predictions require large context, but precise predictions require high resolution. Due to memory limitations in current hardware, previous approaches tend to take low resolution images as input to cover large spatial context, and produce less precise (or low resolution) 3D estimates as a result. We address this limitation by formulating a multi-level architecture that is end-to-end trainable. A coarse level observes the whole image at lower resolution and focuses on holistic reasoning. This provides context to an fine level which estimates highly detailed geometry by observing higher-resolution images. We demonstrate that our approach significantly outperforms existing state-of-the-art techniques on single image human shape reconstruction by fully leveraging 1k-resolution input images.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Sparse Voxel Fields (2020, background): “…point (Michalkiewicz et al., 2019; Mescheder et al., 2019; Chen and Zhang, 2019; Park et al., 2019; Peng et al., 2020), while other works learn 3D representations only from images with differentiable renderers (Liu et al., 2019d; Saito et al., 2019, 2020; Niemeyer et al., 2019; Jiang et al., 2020).”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors (2021, background): “PIFuHD [38] extended PIFu to capture more local details.”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “Recently, [41, 47, 60, 48] capture the human prior from training data using networks, which enables them to recover 3D human geometry and texture from a single image.”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “In contrast to other work which approximates hair with textures or coarse geometry [17, 55], we represent hair as individual 3D strands, with a full head of hair comprising over 100,000 strands.”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “Recent works, [20, 47, 48, 66, 61] utilize sampled perpixel encodings similar to us.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “The success of deep implicit representations in general object modeling has inspired research in 3D human reconstruction [8], [54], [55], [56], [57].”
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes (2021, background): “PIFu [38, 39] learns features on a dense 2D grid with depth as an additional input parameter, while other works learn these on sparse regular [15, 3, 20] or deformed [13] 3D grids.”
Advances in Neural Rendering (2021, methodology): “The method also favourably compares to human mesh reconstruction techniques such as PIFuHD [SSSJ20], which strongly depends on training 3D data when it comes to the 3D reconstruction of ﬁne appearance details (e.g., rarely-worn or unique garments).”
Neural RGB-D Surface Reconstruction (2021, background): “[60] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.”
Learning to Recover 3D Scene Shape from a Single Image (2020, background): “PIFu [30, 31] proposes an memory-efficient implicit function to recover high-resolution surfaces, including unseen/occluded regions, of humans.”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, background): “For example, researchers have explored to represent human body using a 3D mesh [20, 7], a volumetric space [47], or an occupancy field [41, 42].”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, background): “Another two PIFu variations are PIFuHD [22] and ARCH [8].”
Learning Continuous Image Representation with Local Implicit Image Function (2020, methodology): “Despite its success in 3D tasks [38, 39], previous encoderbased methods of implicit neural representation only succeeded in representing simple images such as digits [6], but failed to represent natural images with high fidelity [41].”
Pixel-aligned Volumetric Avatars (2021, background): “PIFu/PIFuHD [22, 23] models human bodies with an implicit function evaluated at the depth of a point.”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “To evaluate performance without a domain gap between train/test data, we also test all models on “AGORA-50” [46, 47], which contains 50 samples from AGORA that are different from the 450 used for training.”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, methodology): “This representation has been applied successfully to shape representation [26, 27, 31, 4, 1, 10, 47, 32], novel view synthesis [38, 23, 15, 28, 21, 33, 34, 40, 37] and multi-view 3D reconstruction [46, 29, 17, 12, 22].”