DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing
By Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui. 2019
310 citations recorded by the source corpus.
We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward pass of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backward to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Jonathan T. Barron
- Songyou Peng
- Michael Niemeyer
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Matthew Tancik
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Michael Niemeyer
- Matthew Tancik
- Songyou Peng
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Andreas Geiger
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Andrea Vedaldi
- Xiaogang Wang
In 3D Vision & Pose
- Andreas Geiger
- Jonathan T. Barron
- Matthias Nießner
- Michael Niemeyer
- Matthew Tancik
- Songyou Peng
- Ben Mildenhall
- Christian Theobalt
Citations in the corpus
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
- Advances in Neural Rendering (2021, background): “An implicit differentiable renderer [SZW19, NMOG20,LZP ∗ 20,LSCL19,YKM ∗ 20,KJJ ∗ 21,BKW21,TLY ∗ 21] can be implemented by first finding the intersection between a viewing ray and the surface using the geometric branch of the neural implicit function, and then obtaining the RGB value of this point fro”
- NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Neural implicit representations have made remarkable achievements in novel view synthesis [53, 33, 22, 35, 30, 52] and 3D/4D reconstruction [47, 48, 66, 37, 23, 20, 32, 39, 59, 65].”
- Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes (2021, methodology): “DIST [27] and SDFDiff [22] present differentiable renderers for SDFs using sphere tracing.”
- Efficient Geometry-aware 3D Generative Adversarial Networks (2021, unknown): “Emerging neural scene representations use differentiable 3D-aware representations [1, 3, 6, 8, 13, 17, 43, 44, 52, 65] that can be optimized using 2D multi-view images via neural rendering [15, 20, 24, 30, 34–37, 40, 45, 46, 50, 51, 54, 62, 63, 70– 72].”
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, methodology): “Recent works represent surfaces as signed distance functions (SDF) [46, 49, 17, 22] or occupancy [29, 30].”
- 3D Neural Field Generation Using Triplane Diffusion (2022, unknown): “…geometry [1,4,5,10,11,15,21,23,45, 47,48,57,65,73] or use posed images to jointly optimize geometry and appearance [6,7,18,25,29,33,36–39, 44, 49, 50, 54, 55 Triplanesareefficient, scalingwiththesurfacearearatherthan volume, and naturally integrate with expressive, fine-tuned 2D generator…”
- IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “With advances in differentiable rendering methods [7, 24, 30, 33, 34, 40], many methods [32, 35, 40, 48, 50, 51, 52, 56, 62] showed the ability to learn both scene geometry and appearance from multi-view observations (though only for simple geometry and diffuse materials).”
- Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, unknown): “In these works, deep neural networks are employed to learn novel view synthesis from 2D images in various approaches, such as voxels [25, 15], point clouds [1, 20], textured meshes [27, 13, 11], multi-plane images [27, 31, 6], and implicit functions [26, 14, 18].”
- Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, background): “Prior work extracts either a global image feature [3, 14, 17, 18], or pixel-aligned features (PIFu [21]) to drive this estimation.”
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, result): “In contrast to the explicit scene representations used by classic MVS algorithms, recent neural approaches [39,48,82] represent surfaces via a single MLP with continuous outputs.”
- GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “To overcome the need of 3D supervision, several works [50, 51, 66, 81, 92] propose differentiable rendering techniques.”
- DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “3D reconstruction from implicit models has undergone significant advancements, largely fueled by the integration of neural networks [61, 72, 81, 148, 173].”
- Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “Later work combined this idea with various formulations of differentiable rendering to recover neural representations of 3D shape using only 2D image observations [18, 19, 24, 26, 34, 37].”