Cost Volume Pyramid Based Depth Inference for Multi-View Stereo

By Jiayu Yang, Wei Mao, José Manuel Álvarez, Miaomiao Liu. Computer Vision and Pattern Recognition, 2019

414 citations recorded by the source corpus.

We propose a cost volume-based neural network for depth inference from multi-view images. We demonstrate that building a cost volume pyramid in a coarse-to-fine manner instead of constructing a cost volume at a fixed resolution leads to a compact, lightweight network and allows us inferring high resolution depth maps to achieve better reconstruction results. To this end, we first build a cost volume based on uniform sampling of fronto-parallel planes across the entire depth range at the coarsest resolution of an image. Then, given current depth estimate, we construct new cost volumes iteratively on the pixelwise depth residual to perform depth map refinement. While sharing similar insight with Point-MVSNet as predicting and refining depth iteratively, we show that working on cost volume pyramid can lead to a more compact, yet efficient network structure compared with the Point-MVSNet on 3D points. We further provide detailed analyses of the relation between (residual) depth sampling and image resolution, which serves as a principle for building compact cost volume pyramid. Experimental results on benchmark datasets show that our model can perform 6x faster and has similar performance as state-of-the-art methods. Code is available at https://github.com/JiayuYANG/CVP-MVSNet

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/73fdd0c9c0a4f6e07fda16449db1fe703c13ef23

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “…the fruits of decades of advances in various sub-fields such as keypoint detection [26, 28, 63, 97] and matching [10, 60, 100, 120], robust estimation [3, 10, 181], Structure-from-Motion (SfM) and Bundle Adjustment (BA) [20, 59, 106], dense Multi-View Stereo (MVS) [40, 107, 139, 158, 176], etc.”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “CVP-MVSNet [41] forms an image pyramid and also constructs a cost volume pyramid.”
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching (2021, background): “Besides, previous work [34] argues that small resolution cost volume is too small to generate a reasonable initial disparity map to be refined.”
MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2024, neutral): “While 3D reconstruction from a set of images has been extensively studied with Structure-from-Motion (SfM) [1, 43] and Multi-View Stereo (MVS) [44, 59] techniques, recovering the 3D geometry from a single image in an arbitrary domain remains a significant challenge due to its highly ill-posed…”
Grounding Image Matching in 3D with MASt3R (2024, neutral): “…0.835 0.554 0.695 Furu [31] 0.613 0.941 0.777 Tola [90] 0.342 1.190 0.766 Gipuma [32] 0.283 0.873 0.578 MVSNet [110] 0.396 0.527 0.462 CVP-MVSNet [109] 0.296 0.406 0.351 UCS-Net [17] 0.338 0.349 0.344 CER-MVS [55] 0.359 0.305 0.332 CIDER [107] 0.417 0.437 0.427 PatchmatchNet [99] 0.427 0.277…”
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers (2020, background): “Following the same idea of computing matching cost by learnt 3D convolutions, other works [40, 7, 41, 15] attempt to enable high resolution inference, mitigate the memory constraint, and/or leverage richer context information via a multi-resolution approach.”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “The cost volume technique has been successfully applied in depth estimation [11, 55, 18] and optical ﬂow estimation [35, 10, 53] for associating pixels between two frames.”