NeRD: Neural Reflectance Decomposition from Image Collections

By Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, Hendrik P. A. Lensch. IEEE International Conference on Computer Vision, 2020

627 citations recorded by the source corpus.

Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model the radiance field of an object, most of these techniques only enable view synthesis and not relighting. Additionally, evaluating these radiance fields is resource and time-intensive. We propose a neural reflectance decomposition (NeRD) technique that uses physically-based rendering to decompose the scene into spatially varying BRDF material properties. In contrast to existing techniques, our input images can be captured under different illumination conditions. In addition, we also propose techniques to convert the learned reflectance volume into a relightable textured mesh enabling fast real-time rendering with novel illuminations. We demonstrate the potential of the proposed approach with experiments on both synthetic and real datasets, where we are able to obtain high-quality relightable 3D assets from image collections. The datasets and code are available at the project page: https://markboss.me/publication/2021-nerd/.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/aa91b25cf28e4ef58aeba19e72b233fb5f97d4ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

DreamFusion: Text-to-3D using 2D Diffusion (2022, background): “Previous work on generative or multiview 3D reconstruction using NeRF-like models have proposed a variety of reflectance models (Bi et al., 2020; Boss et al., 2021; Srinivasan et al., 2021; Pan et al., 2022).”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “We base our rendering model on Spherical Gaussians (SG) [4, 60].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Apart from lighting decomposition discussed in prior work [8, 9, 111], we further investigate the new challenge of smooth volume decomposition by experimenting with different DFF setups.”
Volume Rendering of Neural Implicit Surfaces (2021, background): “Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due to its expressive representation power and low memory foot-print, including scene (geometry & appearance) representation [9, 20, 19, 23, 25, 29, 36, 28, 35] and free-view rendering [33, 16, 30, 26, 17”
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction (2021, methodology): “NeRF [34] and follow-ups [6,29,35,36,44,45,49,53,64] use volume rendering by learning alpha-compositing of a radiance field along rays.”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, result): “Other works [Boss et al. 2021; Kuang et al. 2022; Srinivasan et al. 2021; Zhangetal.2021a,b]attempttodecomposeascene’sview-dependent appearanceintomaterialandlightingproperties.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…through the use of MLPs. Subsequent research has expanded the utility of NeRF to various applications, encompassing tasks such as mesh reconstruction from a collection of images [20, 45], inverse rendering [5, 25, 54], optimization of camera parameters [21, 47, 48], and few-shot learning [10, 51].”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The success of NeRF has also inspired many follow-up works that extend the NeRF to dynamic scenes [30, 33, 8, 40], relighting [3, 39], generative models [37, 25, 4, 13], etc.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “It has also been extended on many fronts, e.g . reﬂectance modeling for photorealistic relighting [3, 50] and dynamic scene modeling that integrates the motion of the world [25, 63, 39].”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “[4, 15, 54] enable learning from images with varying light conditions.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, positive): “Prior NeRF-based models define a normal vector field in the scene by either using the spatial MLP to predict unit vectors [3, 45] at any 3D location, or by using the gradient of the volume density with respect to 3D position [4, 34]:”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”
TensoRF: Tensorial Radiance Fields (2022, background): “This representation has been quickly extended and applied in diverse graphics and vision applications, including generative models [6,26], appearance acquisition [2,4], surface reconstruction [40,27], fast rendering [33,47,12], appearance editing [42,19], dynamic capture [17,28].”