Differentiable Monte Carlo ray tracing through edge sampling

By Tzu-Mao Li, Miika Aittala, Frédo Durand, Jaakko Lehtinen. ACM Transactions on Graphics, 2018

586 citations recorded by the source corpus.

Gradient-based methods are becoming increasingly important for computer graphics, machine learning, and computer vision. The ability to compute gradients is crucial to optimization, inverse problems, and deep learning. In rendering, the gradient is required with respect to variables such as camera parameters, light sources, scene geometry, or material appearance. However, computing the gradient of rendering is challenging because the rendering integral includes visibility terms that are not differentiable. Previous work on differentiable rendering has focused on approximate solutions. They often do not handle secondary effects such as shadows or global illumination, or they do not provide the gradient with respect to variables other than pixel coordinates. We introduce a general-purpose differentiable ray tracer, which, to our knowledge, is the first comprehensive solution that is able to compute derivatives of scalar functions over a rendered image with respect to arbitrary scene parameters such as camera pose, scene geometry, materials, and lighting parameters. The key to our method is a novel edge sampling algorithm that directly samples the Dirac delta functions introduced by the derivatives of the discontinuous integrand. We also develop efficient importance sampling methods based on spatial hierarchies. Our method can generate gradients in times running from seconds to minutes depending on scene complexity and desired precision. We interface our differentiable ray tracer with the deep learning library PyTorch and show prototype applications in inverse rendering and the generation of adversarial examples for neural networks.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0a3f6b49e632917fbea0c63860c14d24143641eb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “[30] introduce a differentiable ray tracer through edge sampling.”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, unknown): “with a differentiable path tracer [29, 42].”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, background): “A few recent works from the graphics community proposed differentiable Monte Carlo renderers [18, 4] for optimizing rendering parameters (e.”
Accelerating 3D deep learning with PyTorch3D (2019, methodology): “Differentiable ray tracing methods, such as Redner [30] and Mitsuba2 [40], give more photorealistic images at the expense of increased compute.”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, background): “An inverse rendering system, or a differentiable renderer [25, 27, 30, 34, 39], provides access to the Jacobian J π ≜ ∂ x π ∂ θ of a rendered image x π at camera viewpoint π with respect to the underlying 3D parameterization θ .”
Differentiable vector graphics rasterization for editing and learning (2020, methodology): “While previous 3D differentiable rendering methods focused on triangle meshes and 3D radiative transfer equations [Li et al. 2018; Zhang et al. 2019], we present a simpler derivation tailored to 2D vector graphics that work with curved boundaries.”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, background): “Some recent work has proposed differentiable general-purpose global illumination renderers [30, 12]; unlike our more specialized, fast rendering layer, these are too expensive to use for neural network training.”
IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “With advances in differentiable rendering methods [7, 24, 30, 33, 34, 40], many methods [32, 35, 40, 48, 50, 51, 52, 56, 62] showed the ability to learn both scene geometry and appearance from multi-view observations (though only for simple geometry and diffuse materials).”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, methodology): “Different from recent work that employs hand-crafted differentiable renderers [18, 22, 29, 34, 36, 51, 64], HoloGAN learns perspective projection and rendering of 3D features from scratch using a projection unit [40].”
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting (2021, methodology): “We also tried redner [19], a Monte Carlo differentiable renderer representing shapes as meshes.”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, background): “However, these 3D representations are all discretized up to a certain resolution, leading to the loss of geometric details and breaking the differentiable properties [23].”
Gradients are Not All You Need (2021, background): “…et al., 2019, Werling et al., 2021, Degrave et al., 2019, de Avila Belbute-Peres et al., 2018, Gradu et al., 2021, Freeman et al., 2021], graphics [Li et al., 2018, Kato et al., 2020], molecular dynamics [Schoenholz and Cubuk, 2020, Hinsen, 2000], differentiating though optimization procedures…”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “More recently, Li et al. [24] introduce a differentiable ray tracer to realize the differentiability of secondary rendering effects.”
Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, background): “Although simple differentiable renderers have been proposed, for instance in [32], they are not yet part of typical game engines.”
DiffTaichi: Differentiable Programming for Physical Simulation (2019, background): “The deep learning community also often incorporates differentiable rendering operations (OpenDR (Loper & Black, 2014), N3MR (Kato et al., 2018), redner (Li et al., 2018a), Mitsuba 2 (Nimier-David et al., 2019)) to learn from 3D scenes.”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “We note that recent work on differentiable rendering [20, 28] could potentially benefit this work.”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, background): “In [17], the authors introduced a differentiable ray tracer which builds on Monte Carlo ray tracing, and can thus deal with secondary lighting effects such as shadows and indirect light.”