Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields

By Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, Pratul P. Srinivasan. Computer Vision and Pattern Recognition, 2021

792 citations recorded by the source corpus.

Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons that provide the volume density and view-dependent emitted radiance at each location. While NeRF-based techniques excel at representing fine geometric structures with smoothly varying view-dependent appearance, they often fail to accurately capture and reproduce the appearance of glossy surfaces. We address this limitation by introducing Ref-NeRF, which replaces NeRF's parameterization of view-dependent outgoing radiance with a representation of reflected radiance and structures this function using a collection of spatially-varying scene properties. We show that together with a regularizer on normal vectors, our model significantly improves the realism and accuracy of specular reflections. Furthermore, we show that our model's internal representation of outgoing radiance is interpretable and useful for scene editing.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/40c8c8d8a41c16a0e017cc0d059fae9d346795f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “Our estimated surface may be incorrect, especially for scenes with specular surfaces and/or sparse views (Figure 8a); it uses binary opacities to avoid sorting polygons, and thus cannot handle scenes with semitransparencies (Figure 8b); it uses fixed mesh and texture resolutions, which may be too co”
DreamFusion: Text-to-3D using 2D Diffusion (2022, methodology): “To prevent pathologies in the density field where normal vectors face backwards away from the camera we use a modified version of the orientation loss proposed in Ref-NeRF (Verbin et al., 2022).”
Instant neural graphics primitives with a multiresolution hash encoding (2022, neutral): “2021] and the spherical harmonics basis in NeRF, similar to concurrent work [Verbin et al. 2021; Yu et al. 2021a].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “The diffuse color and specular tint were multiplied together, and added to the specular color (output of directional MLP) which gave the N e R F [ 1 ] Fundamentals(Sec.3 .1) mip-NeRF [37] based mip-NeRF[37] (2021), Ref-NeRF [38] (2021), RawNeRF (2021) [39] Deformation-based Nerﬁes [40] (2020), Hyper”
Recent advances in 3D Gaussian splatting (2024, neutral): “After texture and lighting disentanglement, these methods can independently modify texture or lighting without influencing the other.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “NeRF [46] and its variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation…”
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, background): “, 2020), have achieved photorealistic static scene rendering (Barron et al., 2021; Verbin et al., 2022).”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “The seminal work, Neural Radiance Fields (NeRF) [21], has been widely studied and advanced in a variety of applications [2, 3, 13, 19, 23, 25, 32], including novel view synthesis [18, 21], 3D generation [10, 25], deformation [23,26,28], video [7,14,15,24,35].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “The development of Neural Radiance Fields (NeRF) [55, 47] has prompted a shift towards reconstructing 3D as volume radiance [83], enabling the synthesis of photo-realistic novel views [85, 3, 4].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, neutral): “One potential future avenue is to specifically address reflections as shown in Ref-NeRF [Verbin et al. 2022], which however is orthogonal to goals of our work.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, background): “Ref-NeRF [Verbin et al. 2022] reparameterizes NeRF’s view-dependent appearance to enable appearance editing and improve the reconstruction and rendering of specular materials.”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, background): “Previously, various representations have been introduce to text-to-3D generations [3, 8, 31, 39, 44].”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, methodology): “2022a], Ref-NeRF [Verbin et al. 2022], and NeRF-W [Martin-Brualla et al.”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “One approach is to impose physics-based inductive biases in its optimization process to enable changes in materials or scene lighting [44, 1, 40, 27, 25].”