Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
By Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, Andreas Geiger. 2019
1,118 citations recorded by the source corpus.
Learning-based 3D reconstruction methods have shown impressive results. However, most methods require 3D supervision which is often hard to obtain for real-world datasets. Recently, several works have proposed differentiable rendering techniques to train reconstruction models from RGB images. Unfortunately, these approaches are currently restricted to voxel- and mesh-based representations, suffering from discretization or low resolution. In this work, we propose a differentiable rendering formulation for implicit shape and texture representations. Implicit representations have recently gained popularity as they represent shape and texture continuously. Our key insight is that depth gradients can be derived analytically using the concept of implicit differentiation. This allows us to learn implicit shape and texture representations directly from RGB images. We experimentally show that our single-view reconstructions rival those learned with full 3D supervision. Moreover, we find that our method can be used for multi-view 3D reconstruction, directly resulting in watertight meshes.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Jonathan T. Barron
- Michael Niemeyer
- Gerard Pons-Moll
- Matthias Nießner
- Ben Mildenhall
- Songyou Peng
- Pratul P. Srinivasan
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Matthew Tancik
- Peter Hedman
- Michael Niemeyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Ming-Hsuan Yang
- Dacheng Tao
- Andrea Vedaldi
- Trevor Darrell
- Xiaogang Wang
- Andrew Zisserman
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Jonathan T. Barron
- Matthias Nießner
- Jitendra Malik
- Justin Johnson
- Matthew Tancik
- Andrea Vedaldi
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “We call this detailed conditioning and compare it with two popular choices: one where a globally average-pooled vector is used, as in [37, 42], and one where the feature vector is bilinearly interpolated at the projected location in the feature map, as in [21, 47, 78].”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
- D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, methodology): “[34, 57] introduced an implicit-based method to calculate the exact derivative of a 3D occupancy field surface intersection with a camera ray.”
- Kubric: A scalable dataset generator (2022, background): “Consider the core vision problem of reconstructing a 3D scene from few observations [68, 86, 87, 114].”
- 3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “They either solely learn geometry [1,4,5,10,11,15,21,23, 44, 46, 47, 56, 64, 72] or use posed images to jointly optimize geometry and appearance [6, 7, 18, 25, 28, 32, 35– 38, 43, 48, 49, 53, 54, 58, 66, 71, 81–83].”
- Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, positive): “In our experiments, we use a derived dataset [23] that includes object masks and was assembled in the context of prior work [14, 25].”
- NeRV: Neural Representations for Videos (2021, methodology): “It has been widely applied in many 3D vision tasks, such as 3D shapes [9, 10], 3D scenes [11–14], and appearance of the 3D structure [4, 15, 16].”
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “Several works [39,57,67] propose to learn continuous shape and texture representations from posed multi-view images by making the rendering process differentiable.”
- Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Besides using 3D ground truth directly, authors have considered using videos [36], [45], [46], [47], [48], stereo pairs [38], [49] and multi-view images [50], [51], [52], [53], [54].”
- NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “representation methods [8][9][10], the original NeRF model converged to better results in terms of visual quality, with more recent models performing even better.”
- Convolutional Occupancy Networks (2020, methodology): “We use the network architecture of [29], adding ψ to the input features of every ResNet block instead of the more memory intensive batch normalization operation proposed in earlier works [26].”
- PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, methodology): “These networks have been used to predict occupancy [26, 4, 32, 38, 29, 19], signed distance fields [30, 10, 59, 60], and radiance [28].”
- NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Neural implicit representations have made remarkable achievements in novel view synthesis [53, 33, 22, 35, 30, 52] and 3D/4D reconstruction [47, 48, 66, 37, 23, 20, 32, 39, 59, 65].”
- Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, neutral): “In these works, deep neural networks are employed to learn novel view synthesis from 2D images in various approaches, such as voxels [25, 15], point clouds [1, 20], textured meshes [27, 13, 11], multi-plane images [27, 31, 6], and implicit functions [26, 14, 18].”
- Neural Scene Graphs for Dynamic Scenes (2020, methodology): “Adding generalization across a class of objects can also be seen in other recent work that either use an explicit encoder network to predict a latent code from an image [20, 24], or use a latent code to predict network weights directly [32].”
- NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis (2020, background): “The resulting Neural Reflectance and Visibility Field (NeRV) is visualized in Figure 3.”
- pixelNeRF: Neural Radiance Fields from One or Few Images (2020, methodology): “The majority employs global image features [29, 6, 28, 25, 8], which, while memory efficient, cannot preserve details that are present in the image and often lead to retrieval-like results.”
- Implicit Diffusion Models for Continuous Super-Resolution (2023, positive): “In recent years, implicit neural representations have shown extraordinary capability in modeling 3D object shapes, synthesizing 3D surfaces of the scene, and capturing complicated 3D structures [3, 27–29 , 36– 38].”