Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference

By Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, Long Quan. Computer Vision and Pattern Recognition, 2019

708 citations recorded by the source corpus.

Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to high-resolution scenes. In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU). This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at https://github.com/YoYo000/MVSNet.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2e54e38db2b276934d6afce9a0ee9dfae0b03f33

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “, feature matching [22,35,38,70,79], depth fusion [15,54], or inferring depth from multi-view images [28,73,74,78].”
Free View Synthesis (2020, methodology): “We also experimented with more recent MVS methods [46,47], but did not observe significant improvements in the mapping and blending performance on large scale scenes.”
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (2021, methodology): “Following works have extended this technique with recurrent plane sweeping [51], point-based densification [8], confidence-based aggregation [30], and multiple cost volumes [10, 18], improving the reconstruction quality.”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “This helps our model be applicable to complex and large-scale scenes [39, 43].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, methodology): “The first stage uses a small standard plane sweep volume with low image resolution and relatively sparse depth planes – 64 planes that are fewer than the number of planes (256 or 512) in previous work [58, 59]; the following two stages use ATVs with higher image resolutions and significantly fewer d”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “Based on MVSNet, RMVSNet (Yao et al. 2019) uses GRU to reduce memory cost, MVSCRF (Xue et al.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “Recently, deep learning-based MVS methods have also been proposed [16,25,53,69,70], offering efficient high-quality reconstruction in a feed-forward manner.”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “As mentioned in [4,44,45], an increased number of plane hypothesis D, a larger spatial resolution W × H , and a finer plane interval are likely to improve the reconstruction accuracy.”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “While the problem is previously addressed by classical methods, recent studies [30, 31, 10] show that learning-based approaches are also able to produce results comparable to or even better than classical state-of-the-arts.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, background): “Yao et al. [92] sequentially regularize the 2D cost maps along the depth direction via a Gated Recurrent Unit (GRU), see Fig.”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, methodology): “Common approaches usually build a single cost volume at a fixed resolution [39, 40, 14], which incurs in large memory requirements and thus, limits the use of high-resolution images.”
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction (2021, background): “For instance, some works learn to match 2D features [15, 21, 26, 56, 62], fuse depth maps [11, 46], or infer depth maps [16, 58, 59] from multi-view images.”