MVSNet: Depth Inference for Unstructured Multi-view Stereo
By Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, Long Quan. 2018
1,670 citations recorded by the source corpus.
We present an end-to-end deep learning architecture for depth map inference from multi-view images. In the network, we first extract deep visual image features, and then build the 3D cost volume upon the reference camera frustum via the differentiable homography warping. Next, we apply 3D convolutions to regularize and regress the initial depth map, which is then refined with the reference image to generate the final output. Our framework flexibly adapts arbitrary N-view inputs using a variance-based cost metric that maps multiple features into one cost feature. The proposed MVSNet is demonstrated on the large-scale indoor DTU dataset. With simple post-processing, our method not only significantly outperforms previous state-of-the-arts, but also is several times faster in runtime. We also evaluate MVSNet on the complex outdoor Tanks and Temples dataset, where our method ranks first before April 18, 2018 without any fine-tuning, showing the strong generalization ability of MVSNet.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Andreas Geiger
- Marc Pollefeys
- Vladlen Koltun
- Matthias Nießner
- Jia Deng
- Zachary Teed
- Torsten Sattler
- Jia-Bin Huang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Jitendra Malik
- Andreas Geiger
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Songyou Peng
- Matthias Nießner
- Yaron Lipman
- Marc Pollefeys
- Lars M. Mescheder
- Matan Atzmon
Citations in the corpus
- 3D Instance Segmentation of MVS Buildings (2021, methodology): “Multi-view images are captured by Unmanned Aerial Vehicle (UAV) and used to automatically reconstruct dense 3D mesh models of large urban scenes [1], [2].”
- AANet: Adaptive Aggregation Network for Efficient Stereo Matching (2020, background): “, high-resolution stereo matching [37], multi-view stereo [38] and optical flow estimation [30].”
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “Typically, these methods downsample the feature maps to formulate the cost volumes at a lower resolution [3,4,8,12,15,22,26,38,44,45,48], and adopt upsampling [3, 8, 15, 26, 34, 38, 41, 48] or postrefinement [4, 22] to output the final high-resolution result.”
- pixelNeRF: Neural Radiance Fields from One or Few Images (2020, critical): “We found that the standard data split introduced in MVSNet [51] contains overlap between scenes of the training and test sets.”
- Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “Most recently, deep learning based MVS models have drawn attention, and most of these approaches [50, 15, 17, 52] rely on a cost volume built from depth hypotheses or plane sweeps.”
- Learning the Depths of Moving People by Watching Frozen People (2019, background): “Some recent learning based methods also consider multiple images, either assuming known camera poses [12, 47] or simultaneously predicting camera poses along with depth [39, 51].”
- Free View Synthesis (2020, methodology): “We also experimented with more recent MVS methods [46,47], but did not observe significant improvements in the mapping and blending performance on large scale scenes.”
- OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Estimating geometry, in the form of scene depth or surface normals, has been widely studied [17, 7, 57, 37].”
- FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “In particular, Reiser et al. [38] accelerate NeRF’s training by splitting a big MLP into thousands of tiny MLPs. MVSNeRF [8] constructs a 3D cost volume [19, 56] and renders high-quality images from novel viewpoints.”
- Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, methodology): “Other methods learn directly the stereo disparity [17, 18, 43].”
- MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, methodology): “Despite the conceptual similarity with the well-established MVS reconstruction pipeline [13,31,48], our approach possesses unique advantages.”
- Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction (2018, methodology): “MVSNet [31] proposes a variance-based approach to capture the second moment information for multiple feature aggregation.”
- Unifying Flow, Stereo and Depth Estimation (2022, methodology): “Different network architectures have been proposed for different tasks in the last few years (e.g, FlowNet [11] for optical flow and MVSNet [12] for multi-view stereo).”
- Neuralangelo: High-Fidelity Neural Surface Reconstruction (2023, background): “Recent learning-based approaches augment the point cloud generation process with learned image features and cost volume construction [2, 10, 40].”
- BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “Based on MVSNet [34], RMVSNet [35] uses GRU to reduce memory cost, MVSCRF [36] adds CRF module, cascade MVSNet [37] Change MVSNet to cascade structure.”
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “Recently, deep learning-based MVS methods have also been proposed [16,25,53,69,70], offering efficient high-quality reconstruction in a feed-forward manner.”
- Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “Recently, learning-based approaches [16, 29, 39, 58, 63, 86, 87] have been proposed that either learn to match image features [39], refine or fuse depth maps [16, 63], optimize parts of the classical MVS pipeline [57], or replace the entire MVS pipeline with neural networks that are trained end-to-e”
- DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, methodology): “In particular, MVSNet (Yao et al., 2018) estimates depth from multiple images.”
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “, feature matching [22,35,38,70,79], depth fusion [15,54], or inferring depth from multi-view images [28,73,74,78].”
- Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Existing multi-view stereo (MVS) methods (either geometric-based [60, 18] or learning-based [24, 72, 20, 37]) assume static scene and thus not suitable for dynamic videos as it often produces erroneous depth for moving objects or untextured regions.”