Local Deep Implicit Functions for 3D Shape

By Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, Thomas A. Funkhouser. 2019

457 citations recorded by the source corpus.

The goal of this project is to learn a 3D shape representation that enables accurate surface reconstruction, compact storage, efficient computation, consistency for similar shapes, generalization across diverse shape categories, and inference from depth camera observations. Towards this end, we introduce Local Deep Implicit Functions (LDIF), a 3D shape representation that decomposes space into a structured set of learned implicit functions. We provide networks that infer the space decomposition and local deep implicit functions from a 3D mesh or posed depth image. During experiments, we find that it provides 10.3 points higher surface reconstruction accuracy (F-Score) than the state-of-the-art (OccNet), while requiring fewer than 1\% of the network parameters. Experiments on posed depth image completion and generalization to unseen classes show 15.8 and 17.8 point improvements over the state-of-the-art, while producing a structured 3D representation for each input with consistency across diverse shape collections.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, methodology): “Regular Grid NKF [Williams et al. 2022] LDIF [Genova et al. 2020] Point2Surf [Erler et al. 2020] DCC-DIF [Li et al. 2022] 3DILG [Zhang et al. 2022] POCO [Boulch and Marlet 2022] Spaghetti [Hertz et al. 2022] Multiple Irregular Grid NeuForm[Lin et al. 2022] Multiple Global Ours”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, background): “3D reconstruction A vast amount of methods studied fully supervised 3D reconstruction from 2D images making use of several different representations: voxel grids [10, 18], meshes [19, 67], point clouds [14, 73], signed distance fields, [49, 2] or continuous occupancy fields [42, 8, 17, 16].”
3D Neural Field Generation Using Triplane Diffusion (2022, background): “Recent years have shown a trend towards locally conditioned representations, which either learn local functions [5,9,27,61] or locally modulate a shared function with a hybrid explicit–implicit representation [4, 6, 12, 19–21, 36, 42, 44, 57].”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “Recently, the community has focused on using implicit functions such as the Signed Distance Function (SDF) [3], Occupancy Networks [2], Implicit Field [65], and their derivations [66, 67], as these can model arbitrary object topology with adjustable resolution.”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “Continuous and differentiable functions parameterized by fully-connected networks (also known as multilayer perceptron, or MLPs) have been successfully applied as compact, implicit representations for modeling 3D shapes [10, 55, 38, 17, 16], object appearances [36, 34], 3D scenes [48, 32, 40].”
IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “Earlier work [2, 14, 23, 36, 41, 53, 61] showed that MLPs can be used as implicit shape representations, where the weights of the MLPs map continuous spatial coordinates to signed distance or occupancy values.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, methodology): “They have been greatly used in 3D shape and geometry modeling [12, 15, 21, 29, 35, 44, 57].”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “Multi-Layer Perceptrons (MLPs) have first been employed for modeling 3D shapes based on signed distance [Chabra et al. 2020; Jiang et al. 2020; Park et al. 2019; Saito et al. 2019a,b] and occupancy fields [Genova et al. 2020; Mescheder et al. 2019; Peng et al. 2020].”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
Shape As Points: A Differentiable Poisson Solver (2021, methodology): “Neural implicit representations have been successfully used to represent geometries of objects [10,16,18,37,42,46,49,52,60,62,65,66] and scenes [8,28,36,48,54,60].”
Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “These methods represent shapes as implicit surfaces defined as a level set of an MLP network and enable full object reconstruction from incomplete 3D point cloud data or depth scans [3, 4, 9, 10, 14, 22, 23, 27].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “representation methods [8][9][10], the original NeRF model converged to better results in terms of visual quality, with more recent models performing even better.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The past few years have witnessed tremendous progress in the implicit representation of 3D models with neural networks [9, 14, 16, 23, 27, 34].”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Such representations have been explored for modeling shape parts [16, 17], objects [3, 6, 10, 19, 25, 27, 30, 34, 35, 40, 42, 43, 49, 53, 60], or scenes [14, 21, 29, 37, 45, 51].”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Emerging neural implicit scene representations promise 3D-structureaware, continuous, memory-efficient representations for parts [13, 12], objects [46, 38, 1, 16, 66, 7, 5], or scenes [10, 57, 20, 47, 56].”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “[44, 1] label each 3D point with its signed distance to the nearest surface point, [35, 6] predict binary per-point occupancies, [11, 10] proposed more structured occupancy functions, and [13, 60] reconstruct meshes from single views.”