Convolutional Occupancy Networks

By Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, Andreas Geiger. European Conference on Computer Vision, 2020

1,136 citations recorded by the source corpus.

Recently, implicit neural representations have gained popularity for learning-based 3D reconstruction. While demonstrating promising results, most implicit approaches are limited to comparably simple geometry of single objects and do not scale to more complicated or large-scale scenes. The key limiting factor of implicit methods is their simple fully-connected network architecture which does not allow for integrating local information in the observations or incorporating inductive biases such as translational equivariance. In this paper, we propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We investigate the effectiveness of the proposed representation by reconstructing complex geometry from noisy point clouds and low-resolution voxel representations. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5a6732513a1dc0bea059543f208a7556e3e31067

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Volume Rendering of Neural Implicit Surfaces (2021, background): “Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due to its expressive representation power and low memory foot-print, including scene (geometry & appearance) representation [9, 20, 19, 23, 25, 29, 36, 28, 35] and free-view rendering [33, 16, 30, 26, 17”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, background): “3D Scene Reconstruction: 3D reconstruction [41, 48, 47, 40, 70, 17, 5, 37, 49, 6] is a traditional but important topic in computer vision.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “This approach has also been applied to compress detailed 3D shapes into neural signed distance functions [61] and to represent large scenes using occupancy networks [45].”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “Michalkiewicz et al. (2019); Mescheder et al. (2019); Park et al. (2019); Peng et al. (2020) predict neural implicit ﬁelds with 3D supervision.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “However, these models adopt triplane NeRF [7,44] as the scene representation, which suffers from a limited triplane resolution and expensive volume rendering.”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “Inspired by [65, 56, 49], we choose the SparseConvNet [21] to efficiently process the structured latent codes, whose architecture is described in Table 1.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Such representations have been explored for modeling shape parts [16, 17], objects [3, 6, 10, 19, 25, 27, 31, 35, 36, 41, 43, 44, 51, 55, 62], or scenes [14, 21, 30, 38, 47, 53].”
IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “Earlier work [2, 14, 23, 37, 42, 44, 66] showed that MLPs can be used as implicit shape representations, where the weights of the MLPs map continuous spatial coordinates to signed distance or occupancy values.”
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors (2021, background): “Fueled by the rapid developments in neural 3D representations(e.g., [34, 35, 6, 29] etc.), a lot of data-driven methods for 3D human reconstruction have been proposed in recent years.”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “Moreover, GANs can be difficult to train and suffer from issues like mode collapse [75] whereas diffusion models train stably and have been shown to better capture the full training distribution.”
MetaSDF: Meta-learning Signed Distance Functions (2020, background): “Parameterized as multilayer perceptrons, these continuous representations have applications in modeling shape parts [10, 11], objects [1, 12–14], or scenes [8, 15, 16].”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “[55] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.”
Neural Sparse Voxel Fields (2020, background): “Most works require 3D supervision during training to infer the SDF value or the occupancy probability of any 3D point (Michalkiewicz et al., 2019; Mescheder et al., 2019; Chen and Zhang, 2019; Park et al., 2019; Peng et al., 2020), while other works learn 3D representations only from images with dif”
Implicit Neural Representations with Periodic Activation Functions (2020, background): “Locally decoding a discrete voxelgrid into an implicit neural representation [11, 12, 51] similarly enables the representation of fine detail.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Most saliently, such ﬁelds have been proposed to represent 3D geometry [5, 25, 27, 29, 33], appearance [22, 24, 34, 37, 38, 44, 47], and tactile properties [11].”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “They have been extensively used to model the 3D shape of objects [Choy et al. 2016; Kar et al. 2017; Mescheder et al. 2019; Peng et al. 2020; Tulsiani et al. 2017; Wu et al. 2016].”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Similar representations, often referred to as tri-planes , have been explored for surface reconstruction from point clouds [Peng et al. 2020] and generative modelling of 3D scenes [DeVries et al. 2021] or faces [Chan et al. 2022].”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, positive): “Implicit surfaces encoded in neural networks are another popular geometry representation for 3D reconstruction, most notably occupancy networks [38, 41, 43, 46] and neural signed distance functions [23, 44, 61, 64].”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “We further report Normal Consistency for the Replica dataset following [21,42,53,54,68,92] as near-perfect ground truth is available.”