DeepMVS: Learning Multi-view Stereopsis

By Po-Han Huang, Kevin Matzen, Johannes Peter Kopf, Narendra Ahuja, Jia-Bin Huang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

596 citations recorded by the source corpus.

We present DeepMVS, a deep convolutional neural network (ConvNet) for multi-view stereo reconstruction. Taking an arbitrary number of posed images as input, we first produce a set of plane-sweep volumes and use the proposed DeepMVS network to predict high-quality disparity maps. The key contributions that enable these results are (1) supervised pretraining on a photorealistic synthetic dataset, (2) an effective method for aggregating information across a set of unordered images, and (3) integrating multi-layer feature activations from the pre-trained VGG-19 network. We validate the efficacy of DeepMVS using the ETH3D Benchmark. Our results show that DeepMVS compares favorably against state-of-the-art conventional MVS algorithms and other ConvNet based methods, particularly for near-textureless regions and thin structures.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/11ca3e9fc044b2e1afe86b2c659daf9c016f7792

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, methodology): “…been used for optical ﬂow [42, 41, 9, 7, 69, 81, 58], robustness estimation of features [51], visual odometry [35, 81], hand tracking [87], shape from shading [82], semantic segmentation [83, 50, 81, 58], amodal segmentation [52, 24], multi-view stereopsis [47] and human pose estimation [89, 26].”
High Quality Monocular Depth Estimation via Transfer Learning (2018, methodology): “In order to compensate for this issue, we consider the reciprocal of the depth [33, 18] where for the original depth map yorig we define the target depth map y as y = m/yorig where m is the maximum depth in the scene (e.”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “Recently, learning-based approaches [16, 27, 37, 53, 58, 79, 80] have been proposed that either learn to match image features [37], refine or fuse depth maps [16, 58], optimize parts of the classical MVS pipeline [52], or replace the entire MVS pipeline with neural networks that are trained end-to-e”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Existing multi-view stereo (MVS) methods (either geometric-based [60, 18] or learning-based [24, 72, 20, 37]) assume static scene and thus not suitable for dynamic videos as it often produces erroneous depth for moving objects or untextured regions.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, background): “Current supervised approaches, e.g. [37, 80, 91, 38, 57], utilize cost volumes but assume ground truth depth and camera poses are available for training.”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “It requires solving tasks such as learning to do visual 3D scene reconstruction [5], [6], [7], [8], self-localization [9], [10], semantic parsing [11], [12], semantic instance understand-”
AdaBins: Depth Estimation Using Adaptive Bins (2020, background): “In recent years, the use of such architectures have shown great success both in the supervised and the unsupervised setting of the depth estimation problem [15, 41, 21, 48, 1].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “Learning-based MVS methods usually replace some parts of the classic MVS pipeline, e.g., feature matching [23, 36, 40, 74, 88], depth fusion [16,57], or inferring depth from multi-view images [29,79,80,85].”
Consistent video depth estimation (2020, methodology): “Recent learning-based methods [Huang et al. 2018; Im et al. 2019; Kusupati et al. 2019; Ummenhofer et al. 2017; Yao et al. 1https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/ ACM Trans.”
Local light field fusion (2019, unknown): “This is in line with recent work that has shown the benefits of deep learning over traditional stereo for depth estimation [Huang et al. 2018; Kendall et al. 2017].”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “Most recently, deep learning based MVS models have drawn attention, and most of these approaches [50, 15, 17, 52] rely on a cost volume built from depth hypotheses or plane sweeps.”
Learning the Depths of Moving People by Watching Frozen People (2019, background): “Some recent learning based methods also consider multiple images, either assuming known camera poses [12, 47] or simultaneously predicting camera poses along with depth [39, 51].”
Multi-agent Embodied Question Answering in Interactive Environments (2020, background): “However, when it comes to predicted depth data, registration between diﬀerent predicted depth frames, or MVS-based techniques like [11] would be required.”
Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference (2019, methodology): “OctNet [23] and O-CNN [27] exploit the sparsity in 3D data and introduce the octree structure to 3D CNNs. SurfaceNet [14] and DeepMVS [13] apply the engineered divide-and-conquer strategy to the MVS reconstruction.”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, unknown): “Training the model on large-scale synthetic datasets has been extensively studied in semantic segmentation [41, 42, 15, 14, 7, 17, 34, 35, 53], multi-view stereo [18], depth estimation [48], optical flow [40, 19, 21], amodal segmentation [16], and object detection [7, 30].”