NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

By Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan T. Barron. Computer Vision and Pattern Recognition, 2020

703 citations recorded by the source corpus.

We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model’s ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4b266567253da27b2cbdb6bb7e450565032a35ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, methodology): “Coordinate-based neural networks have been used for view synthesis in Scene Representation Networks [45], NeRFs [30], and many NeRF extensions [27, 33, 41, 47].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “Other works [Boss et al. 2021; Kuang et al. 2022; Srinivasan et al. 2021; Zhangetal.2021a,b]attempttodecomposeascene’sview-dependent appearanceintomaterialandlightingproperties.”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “Since then, the NeRF representation has been extended to learning more complex phenomenons like dynamic scene flows [35, 20] and illumination effects [29, 14].”
DreamFusion: Text-to-3D using 2D Diffusion (2022, background): “Previous work on generative or multiview 3D reconstruction using NeRF-like models have proposed a variety of reflectance models (Bi et al., 2020; Boss et al., 2021; Srinivasan et al., 2021; Pan et al., 2022).”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, background): “[4, 15, 54] enable learning from images with varying light conditions.”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “One approach is to impose physics-based inductive biases in its optimization process to enable changes in materials or scene lighting [44, 1, 40, 27, 25].”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Neural radiance fields: NeRF [41] and its extensions [3, 23, 44, 58, 62, 75, 77] enable high quality rendering of novel views of static scenes.”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, neutral): “Prior NeRF-based models define a normal vector field in the scene by either using the spatial MLP to predict unit vectors [3, 45] at any 3D location, or by using the gradient of the volume density with respect to 3D position [4, 34]:”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, neutral): “The NeRF representation has been remarkably successful for view synthesis, and follow-on works have extended NeRF for generative modeling [6, 35], dynamic scenes [22, 31], non-rigidly deforming objects [13, 32], and relighting [2, 38].”