Occupancy Networks: Learning 3D Reconstruction in Function Space
By Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger. Computer Vision and Pattern Recognition, 2018
3,446 citations recorded by the source corpus.
With the advent of deep neural networks, learning-based approaches for 3D reconstruction have gained popularity. However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometry of arbitrary topology. Many of the state-of-the-art learning-based 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricted domain. In this paper, we propose Occupancy Networks, a new representation for learning-based 3D reconstruction methods. Occupancy networks implicitly represent the 3D surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches, our representation encodes a description of the 3D output at infinite resolution without excessive memory footprint. We validate that our representation can efficiently encode 3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both qualitatively and quantitatively, for the challenging tasks of 3D reconstruction from single images, noisy point clouds and coarse discrete voxel grids. We believe that occupancy networks will become a useful tool in a wide variety of learning-based 3D tasks.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
View Paper: https://arxiv.org/abs/1812.03828
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Gerard Pons-Moll
- Michael Niemeyer
- Michael J. Black
- Matthias Nießner
- Jonathan T. Barron
- Christian Theobalt
- Songyou Peng
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Michael Niemeyer
- Pratul P. Srinivasan
- Matthias Nießner
- Matthew Tancik
- Justin Johnson
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
- Andrew Zisserman
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Gerard Pons-Moll
- Matthias Nießner
- Jitendra Malik
- Jonathan T. Barron
- Justin Johnson
- Christian Theobalt
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “MCC adopts an occupancy-based representation, similar to OccNets [36], with an attention mechanism on encoded appearance and geometric cues which allows it to predict in any 3D region, even outside the camera frus-tum, efficiently.”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “…MLPs have been used to represent images [28, 38] (referred to as “compositional pattern producing networks”), volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17,…”
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “Recently, concurrently to our work, binary implicit surface representations were used by [12, 34], where they train deep networks across classes of shapes to classify 3D points as inside or outside of a shape.”
- Learning Implicit Fields for Generative Shape Modeling (2018, positive): “single image superresolution using deep networks, e.g., [12,27], which are trained with low- and high-resolution image pairs. Progressive training [24] is another technique to improve image 1Also see [23,33] for concurrent works that propose similar ideas. quality, and we adopt it in our work to red”
- Implicit Behavioral Cloning (2021, background): “There is also a surge of interest in geometry representation learning in implicit representations [44, 45, 46, 47].”
- PointRend: Image Segmentation As Rendering (2019, background): “Instead, recent works consider more efficient non-uniform representations such as meshes [42, 13], signed distance functions [32], and oc-trees [41].”
- Learning Gradient Fields for Shape Generation (2020, positive): “Recently, implicit representations are gaining increasing popularity, as they are capable of representing shapes with high level of detail [47,39,11,40].”
- Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “, as the decision boundary of a binary classifier [12,48], they do not discretize space and have a fixed memory footprint.”
- Learning Shape Templates With Structured Implicit Functions (2019, methodology): “We argue that our structured implicit representation is superior for template learning compared to decoding voxels [6, 55, 56], sparse-voxel octrees [50], points [11], meshes [15, 22, 53], box primitives [51], signed-distance function estimators [40], or indicator function estimators [34].”
- PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020, methodology): “Furthermore, unlike implicit surface representations using a global feature vector [29, 32, 10], PIFu utilizes fully convolutional image features, retaining local details present in an input image.”
- I M Avatar: Implicit Morphable Head Avatars from Videos (2021, background): “While triangulated meshes remain the dominant representation for 3D facial geometry, the recent success of neural implicit shape representations [11, 25, 28, 31, 33, 59] has inspired a number of works that build 3D facial models within this new paradigm.”
- Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “Similar to previous methods [37, 40, 35], we exploit the representational power of deep neural networks and define Ψ as a deep multi-layer perceptron (MLP): (c, σ) = Ψnr(x, r, z).”
- Vector Neurons: A General Framework for SO(3)-Equivariant Networks (2021, methodology): “We build an encoder-decoder framework with the architecture similar to [23] but in the language of VN.”
- GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “Neural representations, as highlighted in prior works [9,62–64, 69, 84, 86 ], present a promising foundation 3D Generation.”
- Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. in the direction of open-world 3D reconstruction by pre-training on large-scale, diverse datasets such as CO3D [44, 31, 36, 16].”
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86, 87].”
- Learning Continuous Image Representation with Local Implicit Image Function (2020, methodology): “How do we represent an image as a continuous function? Our work is inspired by the recent progress in implicit neural representation [34, 27, 6, 38, 18, 41] for 3D shape reconstruction.”
- Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “More recently, several papers start to focus on mechanisms for learning continuous Signed Distance Functions [39], [43] or continuous occupancy grids [157], which are less demanding in terms of memory requirement.”
- Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, positive): “Implicit surfaces encoded in neural networks are another popular geometry representation for 3D reconstruction, most notably occupancy networks [38, 41, 43, 46] and neural signed distance functions [23, 44, 61, 64].”