PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
By Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa. 2019
1,437 citations recorded by the source corpus.
We introduce Pixel-aligned Implicit Function (PIFu), an implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu produces high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.
Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Gerard Pons-Moll
- Michael Niemeyer
- Michael J. Black
- Christian Theobalt
- Jonathan T. Barron
- Songyou Peng
- Matthias Nießner
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
- Andrew Zisserman
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Gerard Pons-Moll
- Jonathan T. Barron
- Jitendra Malik
- Matthias Nießner
- Christian Theobalt
- Angjoo Kanazawa
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Implicit 3D representations such as SDFs [43,52] and occupancy nets (OccNets) [36] have proven effective 3D representations.”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, neutral): “Coordinate-based MLPs have been used to represent images [28, 38] (referred to as “compositional pattern producing networks”), volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, ”
- Implicit Behavioral Cloning (2021, background): “There is also a surge of interest in geometry representation learning in implicit representations [44, 45, 46, 47].”
- Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “Supervision: Recent works [12, 48, 54, 56, 66] have shown that it is possible to learn fθ and tθ with 3D supervision (i.”
- Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, background): “Recently, [41, 47, 60, 48] capture the human prior from training data using networks, which enables them to recover 3D human geometry and texture from a single image.”
- Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Clothes parsing [277] [212] and pose transfer [124] make it possible by inferring the 3D appearance of a person wearing speciic clothes.”
- PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, neutral): “In Table 3 of the main manuscript, the results are reported using mean AP, and variants of AP including AP 50 (AP at OKS = 0.50), AP 75 (AP at OKS = 0.75), AP M for persons with medium sizes, and AP L for persons with large sizes.”
- Advances in Neural Rendering (2021, methodology): “PiFU [SHN ∗ 19] uses an image encoder to compute features on the input image and conditions a 3D MLP on these features via projecting 3D coordinates on the image plane - however, PiFU did not feature a differentiable renderer, and so required ground-truth 3D supervision.”
- Learning to Recover 3D Scene Shape from a Single Image (2020, background): “PIFu [30, 31] proposes an memory-efficient implicit function to recover high-resolution surfaces, including unseen/occluded regions, of humans.”
- Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, neutral): “appearance [22, 24, 34, 37, 38, 44, 47], and tactile properties [11].”
- SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, background): “3D Scene Reconstruction: 3D reconstruction [41, 48, 47, 40, 70, 17, 5, 37, 49, 6] is a traditional but important topic in computer vision.”
- HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “[80] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.”
- Pixel-aligned Volumetric Avatars (2021, methodology): “Inspired by insights from NeRF and PIFu, we demonstrate a framework that handles multiple identities by relying on pixelaligned features.”
- NASA: Neural Articulated Shape Approximation (2019, methodology): “We conclude by clearly contrasting our method, targeting the modeling of O(x|θ) to those that address shape completion O(x|D) [18,56,42,2].”
- Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, background): “Recent work instead leverages local conditioning, where a neural field is conditioned on local features extracted from a context image [26, 6, 27].”
- Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, positive): “Similar to us, [52, 75, 26, 60] utilize per-pixel warp-conditioned embedding [26].”