RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

By Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, Noha Radwan. Computer Vision and Pattern Recognition, 2021

843 citations recorded by the source corpus.

Neural Radiance Fields (NeRF) have emerged as a powerful representation for the task of novel view synthesis due to their simplicity and state-of-the-art performance. Though NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available, its performance drops significantly when this number is reduced. We observe that the majority of artifacts in sparse input scenarios are caused by errors in the estimated scene geometry, and by divergent behavior at the start of training. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints, and annealing the ray sampling space during training. We additionally use a normalizing flow model to regularize the color of unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but in many cases also conditional models that are extensively pre-trained on large multi-view datasets.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7163d171d4671ab8c0fd342e5280db532700999a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “We add a smooth depth loss from RegNeRF [34] and a bilateral normal smoothness loss [5] to encourage smooth 3D surfaces where the projected image gradients are low.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Pre-trained vision models have been used for regularizing NeRF via flows [62], multi-view consistency [35], perceptual loss [110], or depth estimation [100, 77].”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Techniques such as RegNeRF [28], which uses geometry loss from depth patches, and SinNeRF [49], RealFusion [22], and NeuralLift [50], which combine depth maps or Score Dis-tillation Sampling during NeRF training, represent significant steps forward.”
Novel View Synthesis with Diffusion Models (2022, methodology): “Today, these models have been outperformed by NeRF-class models (Yu et al., 2021; Niemeyer et al., 2021; Jang & Agapito, 2021), where 3D consistency is guaranteed by construction , as images are generated by volume rendering of a single underlying 3D representation (a.k.a. “geometry-aware” models).”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, methodology): “Directly applying NeRFs to few-shot scenarios suffers from dramatic degradation [29].”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “…inception, various works have been proposed to enhance performance in diverse scenarios, such as different resolutions of reconstruction [2, 3], the reduced number of training samples [32, 36, 42, 44, 47], and reconstruction of large realistic scenes [29, 41] and dynamic scenes [12, 23, 24, 33].”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Niemeyer et al. [32] proposes a depth smoothness loss as geometry regularization to stabilize training.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, neutral): “Several works proposed to address these issues through regularization using geometric priors [10, 38, 43, 48, 54, 63, 70, 75], which, however, are sensitive to noise in the estimated depth or normals.”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “, depth smoothness [43], surface smoothness [46, 82], semantic similarity [29], or Manhattan world assumptions [20].”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “Also, an Average Error (AVGE) [27] is reported by the geometric mean of MSE = 10 − PSNR / 10 , √ 1 − SSIM, and LPIPS.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “Certain depth regularization [21] [22] [23] [24] methods use the expected depth to restrict densities to delta-like functions at scene surfaces, or to enforce depth smoothness.”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, background): “…., Scene Representation Networks (SRN) [34], Neural Radiance Fields (NeRF) [24] and Light Filed Networks (LFN) [33], these methods are still not satisfactory for practical applications due to expensive per-scene optimization [26,39,43], high memory cost [3,18,44] and slow rendering speed [40,49].”