Local Implicit Grid Representations for 3D Scenes
By Chiyu "Max" Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas A. Funkhouser. 2020
622 citations recorded by the source corpus.
Shape priors learned from data are commonly used to reconstruct 3D objects from partial or noisy data. Yet no such shape priors are available for indoor scenes, since typical 3D autoencoders cannot handle their scale, complexity, or diversity. In this paper, we introduce Local Implicit Grid Representations, a new 3D shape representation designed for scalability and generality. The motivating idea is that most 3D surfaces share geometric details at some scale -- i.e., at a scale smaller than an entire object and larger than a small patch. We train an autoencoder to learn an embedding of local crops of 3D shapes at that size. Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation. We demonstrate the value of this proposed approach for 3D surface reconstruction from sparse point observations, showing significantly better results than alternative approaches.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Michael Niemeyer
- Songyou Peng
- Gerard Pons-Moll
- Jonathan T. Barron
- Matthias Nießner
- Gordon Wetzstein
- Matthew Tancik
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Michael Niemeyer
- Songyou Peng
- Matthew Tancik
- Gordon Wetzstein
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Andreas Geiger
- Jitendra Malik
- Ming-Hsuan Yang
- Dacheng Tao
- Andrea Vedaldi
- Xiaogang Wang
- Trevor Darrell
In Geology
- Matthias Nießner
- Andreas Geiger
- Gerard Pons-Moll
- Michael Niemeyer
- Marc Pollefeys
- Leonidas J. Guibas
- Julian Chibane
- Songyou Peng
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Songyou Peng
- Jonathan T. Barron
- Matthias Nießner
- Gerard Pons-Moll
- Matthew Tancik
- Gordon Wetzstein
Citations in the corpus
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, neutral): “Coordinate-based MLPs have been used to represent images [28, 38] (referred to as “compositional pattern producing networks”), volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, ”
- StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “ey advantages over voxel [7,11, 55,57,70,71] or mesh-based [17,29,47,67] methods are that they do not discretize space and are not restriced in topology. Recent hybrid continuous grid representations [8,22,50] extend implicit representations to complicated or large scale scenes but require 3D input ”
- Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes (2021, background): “PIFu [38, 39] learns features on a dense 2D grid with depth as an additional input parameter, while other works learn these on sparse regular [15, 3, 20] or deformed [13] 3D grids.”
- PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “], [45], [46], [47], [48], [49] have shown promising results on generative models for 3D shape inference based on implicit representations.”
- Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “These methods represent shapes as implicit surfaces defined as a level set of an MLP network and enable full object reconstruction from incomplete 3D point cloud data or depth scans [3, 4, 9, 10, 14, 22, 23, 27].”
- Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction (2020, background): “[27] proposes to use local implicit functions in a grid for detailed 3D reconstruction.”
- Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Local implicit representations [3, 5, 23, 56] and hybrid explicit–implicit representations [11,35,39,53] combine the benefits of both types of representations by offering computationally and memory-efficient architectures.”
- NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, background): “Recently, neural implicit representations [29, 33, 36, 17, 54, 25] have gained significant advances.”
- AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Such representations have been explored for modeling shape parts [16, 17], objects [3, 6, 10, 19, 25, 27, 31, 35, 36, 41, 43, 44, 51, 55, 62], or scenes [14, 21, 30, 38, 47, 53].”
- MetaSDF: Meta-learning Signed Distance Functions (2020, background): “Parameterized as multilayer perceptrons, these continuous representations have applications in modeling shape parts [10, 11], objects [1, 12–14], or scenes [8, 15, 16].”
- CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, methodology): “…existing 3D editing methods op-* erate on explicit 3D representations [46, 15, 7], the recent advances of implicit volumetric representations in capturing and rendering dedicated 3D structures [28, 24, 35, 14, 9, 16] have motivated the research to benefit the manipulation from such representations.”
- pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “These can be supervised with 3D data, such as point clouds, and optimized as either signed distance functions [46, 38, 1, 16, 57, 20, 47, 55, 27] or occupancy networks [37, 6].”
- Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “Multi-Layer Perceptrons (MLPs) have first been employed for modeling 3D shapes based on signed distance [Chabra et al. 2020; Jiang et al. 2020; Park et al. 2019; Saito et al. 2019a,b] and occupancy fields [Genova et al. 2020; Mescheder et al. 2019; Peng et al. 2020].”
- Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “Benefiting from differentiable splatting, we can train the full framework end-to-end on large-scale category-agnostic datasets efficiently (Section 3.3).”
- Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “IFL methods use either binary occupancies [49, 27, 14, 67, 21, 15] or signed distance functions [53, 13, 50, 36] as shape representation for learning.”
- NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Recently, neural implicit representations demonstrated promising results for object geometry representation [8, 22, 24, 32, 34–36, 40, 55, 60, 63, 64], scene completion [6, 17, 37], novel view synthesis [23, 25, 38, 66] and also generative modelling [7, 30, 31, 43].”