Learning a Multi-View Stereo Machine

By Abhishek Kar, Christian Häne, Jitendra Malik. 2017

585 citations recorded by the source corpus.

We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches as well as recent learning based methods.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “The problem of reconstructing 3D geometry from multiple input views has been considered in [31, 35, 52].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “In contrast, [17] is limited to 32(3) voxels and [16] requires processing large volumes using a sliding window, thereby losing global spatial relationships.”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 42, 32, 12, 39, 34], such as decomposing image silhouettes [23? ], Light Field Descriptors [2], and 2D panoramic mapping [38].”
MVSNet: Depth Inference for Unstructured Multi-view Stereo (2018, methodology): “Inspired by recent learning-based stereo [14] and MVS [17,18] methods, we apply the multi-scale 3D CNN for cost volume regularization.”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “LSM [11] and RayNet [21] encoded the camera projection to the network, and utilized 3D CNNs or Markov Random Field to predict surface label.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The past few years have witnessed tremendous progress in the implicit representation of 3D models with neural networks [9, 14, 16, 23, 27, 34].”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “Neural Representations: Besides coordinate-based models, neural representations are commonly used for 3D modeling and rendering [54], ranging from voxel grids [24, 29, 38, 41, 49], neural textures [55], and point clouds [1].”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “Methods such as [8, 7, 46, 55, 24] have demonstrated that, under these restrictive assumptions, it is possible to obtain high-quality reconstructions, motivating researchers to look beyond synthetic data.”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Recently, [40] [41] [42] [43] [44] [45] leverage deep neural nets to learn the 3D shape from multiple RGB images.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Early deep MVS methods used memory-expensive 3D grid representations e.g. [39, 43].”
Learnable Triangulation of Human Pose (2019, methodology): “Specifically unprojection was used in [19] for freepoint-view rendering, in [23, 7] for volumetric 3D object reconstruction.”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, background): “[19] present a learnable system to up-project pixel features to the 3D volume and classify whether a voxel is occupied or not by the surface.”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “Learning-based MVS. Voxel-based methods [20, 21] are restricted to small-scale reconstructions, due to the drawbacks of a volumetric representation.”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, background): “These inductive biases can be explicit representations, such as voxel grids [13, 5, 44], point cloud [7, 24, 18], meshes [41, 43, 14], and implicit representations.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “Kar et al. [60] (Fig.”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “They have been extensively used to model the 3D shape of objects [Choy et al. 2016; Kar et al. 2017; Mescheder et al. 2019; Peng et al. 2020; Tulsiani et al. 2017; Wu et al. 2016].”
DeepMVS: Learning Multi-view Stereopsis (2018, methodology): “In contrast to Recurrent Network-based approaches [3, 18], the use of max-pooling allows us to process an arbitrary number of unordered input images Training deep ConvNets for disparity reconstruction requires a large number of ground truth disparity maps.”
Advances in Neural Rendering (2021, result): “…than realistic image synthesis) adopted an alpha compositing volumetric rendering model in parallel with this view synthesis research [HRRR18,KHM17,TZEM17]; however, these results were heavily constrained by the memory limitations of 3D CNNs and could not produce voxel grid outputs…”
Point-Based Multi-View Stereo Network (2019, background): “Concurrently, 3D cost volume regularization approaches have been proposed [14, 12, 13], where a 3D cost volume is built either in the camera frustum or the scene.”