Point-Based Multi-View Stereo Network

By Rui Chen, Songfang Han, Jing Xu, Hao Su. IEEE International Conference on Computer Vision, 2019

409 citations recorded by the source corpus.

We introduce Point-MVSNet, a novel point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. Our source code and trained models are available at https://github.com/callmeray/PointMVSNet.

Subfields: Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/58a309e2ab84c66450480806e09b8a55f455c218

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “The success of deep-learning methods in computer vision inspired research in this area, using end-to-end deep networks equipped with a correlation cost volume [17, 9, 6, 32], or point-based depth representation and iterative refinement [4].”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “As mentioned in [4,44,45], an increased number of plane hypothesis D, a larger spatial resolution W × H , and a finer plane interval are likely to improve the reconstruction accuracy.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “3D CNN with a recurrent CNN, while P-MVSNet [6] starts with a low resolution MVSNet and then iteratively refines the estimate using their point flow module.”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “We thank Rui Chen for helping train and validate PointMVSNet [4] in our dataset.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “For multi-view input, image features are usually optimized in the constructed 3D geometry volume [28, 29].”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, methodology): “Finally, in contrast to [5] and other related works, our approach can output depth of small resolution with small resolution image.”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “…[20] 0.450 1.040 0.745 MVSNet [42] 0.396 0.527 0.462 R-MVSNet [43] 0.383 0.452 0.417 CIDER [39] 0.417 0.437 0.427 P-MVSNet [28] 0.406 0.434 0.420 Point-MVSNet [6] 0.342 0.411 0.376 Fast-MVSNet [44] 0.336 0.403 0.370 CasMVSNet [17] 0.325 0.385 0.355 UCS-Net [7] 0.338 0.349 0.344 CVP-MVSNet [41] 0”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, methodology): “On the other hand, Point-MVSNet [5] densifies a coarse reconstruction within a predefined local spatial range for better reconstruction with learning-based refinement.”
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (2021, methodology): “Following works have extended this technique with recurrent plane sweeping [51], point-based densification [8], confidence-based aggregation [30], and multiple cost volumes [10, 18], improving the reconstruction quality.”