Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction

By Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven J. Lovegrove, Richard A. Newcombe. European Conference on Computer Vision, 2020

522 citations recorded by the source corpus.

Efficiently reconstructing complex and intricate surfaces at scale is a long-standing goal in machine perception. To address this problem we introduce Deep Local Shapes (DeepLS), a deep shape representation that enables encoding and reconstruction of high-quality 3D shapes without prohibitive memory requirements. DeepLS replaces the dense volumetric signed distance function (SDF) representation used in traditional surface reconstruction systems with a set of locally learned continuous SDFs defined by a neural network, inspired by recent work such as DeepSDF. Unlike DeepSDF, which represents an object-level SDF with a neural network and a single latent code, we store a grid of independent latent codes, each responsible for storing information about surfaces in a small local neighborhood. This decomposition of scenes into local shapes simplifies the prior distribution that the network must learn, and also enables efficient inference. We demonstrate the effectiveness and generalization power of DeepLS by showing object shape encoding and reconstructions of full scenes, where DeepLS delivers high compression, accuracy, and local shape completion.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ce68c019f2b522ac6a4f7fed7c3d3bf26b00d0d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

UniSim: A Neural Closed-Loop Sensor Simulator (2023, background): “Several representations [8, 42, 47, 51, 52, 61, 62,76,100] partition the space and model the volume more efficiently to handle large-scale unbounded outdoor scenes.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Recently, neural implicit representations demonstrated promising results for object geometry representation [8, 22, 24, 32, 34–36, 40, 55, 60, 63, 64], scene completion [6, 17, 37], novel view synthesis [23, 25, 38, 66] and also generative modelling [7, 30, 31, 43].”
Implicit Neural Representations with Periodic Activation Functions (2020, background): “Locally decoding a discrete voxelgrid into an implicit neural representation [11, 12, 51] similarly enables the representation of fine detail.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”
Canonical Capsules: Unsupervised Capsules in Canonical Pose (2020, background): “Recent works have tried to close this gap by using part decomposition consisting of oriented boxes [45], ellipsoids [18, 17], convex polytopes [12], and grids [4].”
Shape As Points: A Differentiable Poisson Solver (2021, methodology): “Neural implicit representations have been successfully used to represent geometries of objects [10,16,18,37,42,46,49,52,60,62,65,66] and scenes [8,28,36,48,54,60].”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, background): “Similar to the local implicit representations [24, 5, 17], the latent codes are used with a neural network to represent the local geometry and appearance of a human.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Such representations have been explored for modeling shape parts [16, 17], objects [3, 6, 10, 19, 25, 27, 30, 34, 35, 40, 42, 43, 49, 53, 60], or scenes [14, 21, 29, 37, 45, 51].”
3D Neural Field Generation Using Triplane Diffusion (2022, unknown): “They either solely learn geometry [1,4,5,10,11,15,21,23,45, 47, 48, 57, 65, 73] or use posed images to jointly optimize geometry and appearance [6,7,18,25,29,33,36–39,44,49,50,54, 55,59,67,72,82–84].”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “It has been proved to be a high efficiency way for modeling 3D objects [35,9,31], 3D scenes [32,43,5,36] and 2D images [17,8,45,42].”
iMAP: Implicit Mapping and Positioning in Real-Time (2021, methodology): “Learning techniques have also been used to improve dense reconstruction but require an existing scan [5] or previous training data [21, 36, 2].”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, unknown): “Methods OccNet [Mescheder et al. 2019] DeepSDF [Park et al. 2019] Single Global IM-Net [Chen and Zhang 2019] ConvOccNet [Peng et al. 2020] IF-Net [Chibane et al. 2020] LIG [Jiang et al. 2020] DeepLS [Chabra et al. 2020] SA-ConvOccNet [Tang et al. 2021]”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Implicit Functions: Using implicit functions to represent 3D geometry has gained popularity in learning-based 3D reconstruction [11, 12, 22, 59, 60, 65, 67, 69, 76] and has been extended to scene-level reconstruction [8, 13, 35, 72, 79].”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Emerging neural implicit scene representations promise 3D-structureaware, continuous, memory-efficient representations for parts [13, 12], objects [46, 38, 1, 16, 66, 7, 5], or scenes [10, 57, 20, 47, 56].”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Compared with explicit representations like voxel and BEV, implicit representations usually share the advantage of arbitrary-resolution modeling and computation-efﬁcient architectures [6, 11, 40].”
Learning Continuous Image Representation with Local Implicit Image Function (2020, methodology): “This idea has been widely applied in modeling 3D object shapes [6, 28, 2, 13], 3D surfaces of the scene [42, 18, 36, 4] as well as the appearance of the 3D structure [33, 32, 29].”