Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion

By Julian Chibane, Thiemo Alldieck, Gerard Pons-Moll. Computer Vision and Pattern Recognition, 2020

568 citations recorded by the source corpus.

While many works focus on 3D reconstruction from images, in this paper, we focus on 3D shape reconstruction and completion from a variety of 3D inputs, which are deficient in some respect: low and high resolution voxels, sparse and dense point clouds, complete or incomplete. Processing of such 3D inputs is an increasingly important problem as they are the output of 3D scanners, which are becoming more accessible, and are the intermediate output of 3D computer vision algorithms. Recently, learned implicit functions have shown great promise as they produce continuous reconstructions. However, we identified two limitations in reconstruction from 3D inputs: 1) details present in the input data are not retained, and 2) poor reconstruction of articulated humans. To solve this, we propose Implicit Feature Networks (IF-Nets), which deliver continuous outputs, can handle multiple topologies, and complete shapes for missing or sparse input data retaining the nice properties of recent learned implicit functions, but critically they can also retain detail when it is present in the input data, and can reconstruct articulated humans. Our work differs from prior work in two crucial aspects. First, instead of using a single vector to encode a 3D shape, we extract a learnable 3-dimensional multi-scale tensor of deep features, which is aligned with the original Euclidean space embedding the shape. Second, instead of classifying x-y-z point coordinates directly, we classify deep features extracted from the tensor at a continuous query point. We show that this forces our model to make decisions based on global and local shape structure, as opposed to point coordinates, which are arbitrary under Euclidean transformations. Experiments demonstrate that IF-Nets outperform prior work in 3D object reconstruction in ShapeNet, and obtain significantly more accurate 3D human reconstructions. Code and project website is available at https://virtualhumans.mpi-inf.mpg.de/ifnets/.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c13375a9bc27bafeeb491fd8becea07d621ac2e7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, methodology): “Speciﬁcally, we train an encoder network similar to [17,18] that takes the segmented and voxelized human S h and object S o point cloud as inputs, and generates a voxel aligned grid of features F = f enc φ ( S h , S o ) .”
GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (2022, background): “Recently, some other works (Luo & Hu, 2021; Chibane et al., 2020) has also been proposed for 3D structure generation with diffusion-based models, but focus on the point cloud problem.”
NASA: Neural Articulated Shape Approximation (2019, background): “There has been a surge of recent interest in representing 3D geometry using implicit functions parameterized by neural networks [37,20,39,10].”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, background): “Firstly, the quality of generation is still lacking and we believe a novel future avenue would be to combine ideas from local implicit methods [8, 45].”
Neural RGB-D Surface Reconstruction (2021, background): “It is related to classical fusion-based 3D reconstruction methods [9, 14, 50, 53, 92], learned 3D reconstruction [7, 15, 47, 58, 79], as well as to recent coordinate-based scene representation models [48, 69, 74].”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “Multi-resolution features grid: To improve the expressiveness and speed of NFFs, past works [12, 51, 74, 98] further combined learnable multi-resolution features grid {G}l=1 with a neural network f .”
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors (2021, methodology): “IFNet [7] and IP-Net [2] can recover impressive 3D humans from partial point clouds, but the dependency on multi-scale 3D convolutions and parametric body models block the realization of real-time reconstruction performance.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “], [45], [46], [47], [48], [49] have shown promising results on generative models for 3D shape inference based on implicit representations.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “for (i) the joint occupancy of the outer (body+clothing) and the inner (body) surfaces and (ii) body part labels. Following recent work IP-Net: Combining Implicit Functions and Parametric Modelling 3 [14], we compute a 3-dimensional multi-scale tensor of deep features from the input point cloud, and”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, methodology): “This interpolation scheme is inspired by DeepVoxels [7, 31] and IF-Net [4].”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, background): “tations for rigid geometry include point-clouds [42,33], voxels [13,48] and octrees [45,39]. Recently, there has been a strong burst in representing 3D data in an implicit manner via a neural network [24,31,6,47,8,12]. The main idea behind this approach is to describe the information (e.g. occupancy”
Local Deep Implicit Functions for 3D Shape (2019, background): “Concurrent work [8, 17] adds a voxel grid structure to deep implicit functions.”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “More speciﬁcally, we investigate the following architectures: a single, large MLP [49,76,81,82], a dense SDF grid [31], a single feature grid [28,38,53,54] and multi-resolution feature grids [10, 22, 45, 69, 92].”
Convolutional Occupancy Networks (2020, methodology): “nt of the camera viewpoint and input representation. Moreover, we demonstrate the feasibility of implicit 3D reconstruction at scene-level as illustrated in Fig.1c. In concurrent work, Chibane et al. [4] present a model similar to our convolutional volume decoder. In contrast to us, they only consid”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “Also, we only compare state-of-the-art deterministic methods: ConvONet [10], IF-Net [2], and PoinTr [14] in these examples.”
Learning Continuous Image Representation with Local Implicit Image Function (2020, background): “Related to recent works [38, 11, 7, 37, 18] on 3D implicit neural representation, LIIF representation is based on local latent codes, which can recover the fine details of natural and complex images.”
Instant neural graphics primitives with a multiresolution hash encoding (2022, methodology): “Second, natural scenes exhibit smoothness, motivating the use of a multi-resolution decomposition [Chibane et al. 2020; Hadadan et al. 2021].”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, background): “Recent implicit function-based meth-ods [45, 40, 9] have exhibited state-of-the-art reconstruction quality.”