Depth-supervised NeRF: Fewer Views and Faster Training for Free

By Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan. 2021

1,154 citations recorded by the source corpus.

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as “free” depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “[8] depth in Abs MSE Abs MSE NeRF-WCE [20] 8.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “Many works address the problem by introducing regularization strategies specified for NeRF [11, 19, 27, 52].”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, background): “Per-scene approaches mainly focus on designing effective regularization terms [8, 11, 26, 39, 43, 50] to better constrain the optimization process.”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, methodology): “However, only few methods show training times speedup, and the improvements are not comparable to ours [1,10,31] or lead to worse quality [6,59].”
Urban Radiance Fields (2021, methodology): “We start by supervising the expected depth ˆ z from a volumetric rendering process (i.e. optical depth [30]) to match the depth of the lidar measurement: Line-of-sight priors .”
Neural RGB-D Surface Reconstruction (2021, background): “Concurrent work that incorporates depth measurements in NeRF focuses on novel view synthesis [16, 49, 81], and uses the depth prior to restrict the volumetric rendering to near-surface regions [49,81] or adds an additional constraint on the depth prediction of NeRF [16].”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, methodology): “A number of techniques are proposed to improve NeRF’s performance with additional regularisation [13, 26, 54], depth priors [7,32,47,53], surface enhancements [27,43,49] or latent codes [40, 44, 52].”
UniDepth: Universal Monocular Metric Depth Estimation (2024, background): “The precise pixel-wise depth estimation is crucial to understanding the geometric scene structure, with applications in 3D modeling [9], robotics [10, 52], and autonomous vehicles [31, 41].”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…through the use of MLPs. Subsequent research has expanded the utility of NeRF to various applications, encompassing tasks such as mesh reconstruction from a collection of images [20, 45], inverse rendering [5, 25, 54], optimization of camera parameters [21, 47, 48], and few-shot learning [10, 51].”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “For instance, external models can be used to acquire normalization-ﬂow regularization [22], perceptual regularization [38], depth supervision [6, 29, 34], and cross-view semantic consistency [11].”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, methodology): “More recent methods [58, 59, 60, 40] have been proposed for constructing NeRF models that generalize across scenes with few input images.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, neutral): “DS-NeRF [10] utilizes sparse depth outputs from Structure-from-Motion (SfM) as supervision.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, neutral): “Several works proposed to address these issues through regularization using geometric priors [10, 38, 43, 48, 54, 63, 70, 75], which, however, are sensitive to noise in the estimated depth or normals.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “A KL divergence term is added to minimize the divergence of the ray’s distribution and this noisy depth distribution (See [22] for details).”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, methodology): “One alternate approach is to optimize the network weights from scratch for every new scene and introduce regularization to improve the performance for sparse inputs, e.g., by adding extra supervision [23] or learning embed-dings representative of the input views [19].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, background): “[Deng et al. 2021] achieves faster training speed with external depth information.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “These unique attributes render both depth and surface normals indispensable in various computer vision applications, including 3D reconstruction [1], [2], [3], neural rendering (NeRF) [4], [5], [6], [7], autonomous driving [8], [9], [10], and robotics [11], [12], [13].”