Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness

By Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Erran L. Li, Ravi Ramamoorthi, Hao Su. Computer Vision and Pattern Recognition, 2019

378 citations recorded by the source corpus.

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes (PSVs) with a fixed depth hypothesis at each plane; this requires densely sampled planes for high accuracy, which is impractical for high-resolution depth because of limited memory. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small PSV to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes with low memory and computation costs; yet, it efficiently partitions local depth ranges within learned small uncertainty intervals. We propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process leads to reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively sub-divides the vast scene space with increasing depth resolution and precision, which enables reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with other learning-based MVS methods on various challenging datasets.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3704411398c7789545a7e3b48d7213cf56ea30e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, background): “Some recent works [12, 4] improve this pipeline with a coarse-to-fine approach.”
Grounding Image Matching in 3D with MASt3R (2024, neutral): “…Furu [31] 0.613 0.941 0.777 Tola [90] 0.342 1.190 0.766 Gipuma [32] 0.283 0.873 0.578 MVSNet [110] 0.396 0.527 0.462 CVP-MVSNet [109] 0.296 0.406 0.351 UCS-Net [17] 0.338 0.349 0.344 CER-MVS [55] 0.359 0.305 0.332 CIDER [107] 0.417 0.437 0.427 PatchmatchNet [99] 0.427 0.277 0.352 GeoMVSNet [119] 0”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “Recently, deep learning-based MVS methods have also been proposed [16,25,53,69,70], offering efficient high-quality reconstruction in a feed-forward manner.”
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching (2021, methodology): “α and β can also be set as hyper parameters, while experiment shows that the learned parameters are more robust than human-selected parameters [4].”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “…[20] 0.450 1.040 0.745 MVSNet [42] 0.396 0.527 0.462 R-MVSNet [43] 0.383 0.452 0.417 CIDER [39] 0.417 0.437 0.427 P-MVSNet [28] 0.406 0.434 0.420 Point-MVSNet [6] 0.342 0.411 0.376 Fast-MVSNet [44] 0.336 0.403 0.370 CasMVSNet [17] 0.325 0.385 0.355 UCS-Net [7] 0.338 0.349 0.344 CVP-MVSNet [41] 0”
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (2021, methodology): “We leverage the variance-based metric to compute the cost, which has been widely used in MVS [50, 10] for geometry reconstruction.”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “Point-based neural rendering methods address this by splatting neural features and using 2D CNNs to render them [2,23,33].”