SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

By Guangcong Wang, Zhaoxi Chen, Chen Change Loy, Ziwei Liu. IEEE International Conference on Computer Vision, 2023

340 citations recorded by the source corpus.

Neural Radiance Field (NeRF) significantly degrades when only a limited number of views are available. To complement the lack of 3D information, depth-based models, such as DSNeRF and MonoSDF, explicitly assume the availability of accurate depth maps of multiple views. They linearly scale the accurate depth maps as supervision to guide the predicted depth of few-shot NeRFs. However, accurate depth maps are difficult and expensive to capture due to wide-range depth distances in the wild. This work presents a new Sparse-view NeRF (SparseNeRF) framework that exploits depth priors from real-world inaccurate observations. The inaccurate depth observations are either from pre-trained depth models or coarse depth maps of consumer-level depth sensors. Since coarse depth maps are not strictly scaled to the ground-truth depth maps, we propose a simple yet effective constraint, a local depth ranking method, on NeRFs such that the expected depth ranking of the NeRF is consistent with that of the coarse depth maps in local patches. To preserve the spatial continuity of the estimated depth of NeRF, we further propose a spatial continuity constraint to encourage the consistency of the expected depth continuity of NeRF with coarse depth maps. Surprisingly, with simple depth ranking constraints, SparseNeRF outperforms all state-of-the-art few-shot NeRF methods (including depth-based models) on standard LLFF and DTU datasets. Moreover, we collect a new dataset NVS-RGBD that contains real-world depth maps from Azure Kinect, ZED 2, and iPhone 13 Pro. Extensive experiments on NVS-RGBD dataset also validate the superiority and generaliz-ability of SparseNeRF. Code and dataset are available at https://sparsenerf.github.io/.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3b0adce2481b4786be8d9810ccb2b1a9dec2f8ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “To tackle the scale ambiguity and error of monocular depths, previous works have introduced various scale-invariant losses [10, 36, 55] including depth ranking loss [42, 50], however, all of which are not optimal for us.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, positive): “Many methods aim to reduce the reliance on dense captures by developing heuristic low-level regularizers based on re-constructed depth [10, 16, 41], visibility [27, 49], appearance [35, 64], or image-space frequencies [65].”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, neutral): “Several works proposed to address these issues through regularization using geometric priors [10, 38, 43, 48, 54, 63, 70, 75], which, however, are sensitive to noise in the estimated depth or normals.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, methodology): “…compared to the previous SparseNeRF [50]. tions [11, 33, 47, 54] within the density field, additional supervision from 2D pre-trained models [23, 50], large-scale pre-training [8, 59], and frequency annealings [55] have been proposed and adopted to address the challenge of few-shot view…”