MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

By Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su. IEEE International Conference on Computer Vision, 2021

1,072 citations recorded by the source corpus.

We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. We train our network on real objects in the DTU dataset, and test it on three different datasets to evaluate its effectiveness and generalizability. Our approach can generalize across scenes (even indoor scenes, completely different from our training scenes of objects) and generate realistic view synthesis results using only three input images, significantly outperforming concurrent works on generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/169971b60749264cbbe2b577dc4d2ad23ca4f46c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, background): “When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “Instead of using optimization-based approaches, we base our reconstruction module on a generalizable SDF reconstruction method SparseNeuS [40], which is essentially a variant of the MVSNeRF [6] pipeline that combines multi-view stereo, neural scene representation, and volume rendering.”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, methodology): “These methods rely on explicit 3D-to-2D projections of samples along the rays to gather features extracted from CNNs, which are then aggregated by a neural network for each point [41, 44, 49], or into a voxel grid [5].”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, methodology): “In the first group, SRF, PixelNeRF, and MVSNeRF are pre-trained in the large-scale DTU datasets (88 scenes) and are directly tested on the LLFF dataset.”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, methodology): “Inspired by classical multi-view stereo, neural networks have also been combined with cost volumes to match features across views [5, 7, 18, 26].”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “However, NeRF still has shortcomings and plenty of work has extended the original NeRF, including better synthesis effects [36, 71, 83], applicable to dynamic scenes [14, 33, 44–48, 67, 72, 82], faster rendering speed [18, 22, 49, 77], generalization to different scenes [8, 69], relighting [4, 60, 8”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, background): “IBRNet [43], MVSNeRF [6], and PixelNeRF [48] construct a generalizable 3D representation by leveraging features gathered from various observed viewpoints.”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “This advantage of Point-NeRF leads to more efficient reconstruction and more accurate rendering than other neural radiance field models [8, 34, 50, 57].”
Urban Radiance Fields (2021, background): “56] or small regions of real scenes [8, 10, 42, 43, 64, 72].”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “…compared to the previous SparseNeRF [50]. tions [11, 33, 47, 54] within the density field, additional supervision from 2D pre-trained models [23, 50], large-scale pre-training [8, 59], and frequency annealings [55] have been proposed and adopted to address the challenge of few-shot view synthesis.”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “Additionally, it in-a spires waves of innovative developments such as NeRF variants [3, 4, 8, 27, 32, 37] and 3D Gaussian Splatting (3DGS) [15], significantly enhancing experiences in virtual reality, augmented reality, and advanced simulations.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Previous methods (Wang et al., 2021; Chen et al., 2021; Long et al., 2022; Reizenstein et al., 2021; Trevithick & Yang, 2021) have tried to learn data priors so as to infer NeRF from a sparse set of images.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, neutral): “Early methods adopted NeRF as the underlying 3D representation [12, 14, 35, 52, 107, 119], but recent work has largely shifted to 3DGS due to its explicit structure and real-time rendering.”
TensoRF: Tensorial Radiance Fields (2022, methodology): “Other methods design generalizable network modules trained across scenes to achieve image-dependent radiance field rendering [48,41,8] and fast reconstruction [7,43].”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “These methods typically leverage 3D-to-2D geometric projection to sample per-view 2D features for 3D reconstruction, using model designs like epipolar lines [57,58,64] or plane-swept cost-volumes [12,29,36] similar to MVS.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…traditional 3D reconstruction methods, such as multi-view stereo or NeRF-based techniques, often demand a dense collection of input images for accurate geometry inference, many of the latest generalizable NeRF solutions [3, 24, 28, 35, 38, 54, 66, 69, 70, 79] strive to learn priors across scenes.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “Some pre-trained methods aim to de-sign a generative model and train it on large datasets [5, 9, 20, 54, 60], while others [16, 49] take pre-trained models as a type of loss to regularize the training process with well-learned knowledge.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “MVSNeRF [4] initializes a plane sweep volume from 3 views before converting it to a NeRF by a pretrained network.”