SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis

By Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, Lu Fang. 2017

455 citations recorded by the source corpus.

This paper proposes an end-to-end learning framework for multiview stereopsis. We term the network SurfaceNet. It takes a set of images and their corresponding camera parameters as input and directly infers the 3D model. The key advantage of the framework is that both photo-consistency as well geometric relations of the surface structure can be directly learned for the purpose of multiview stereopsis in an end-to-end fashion. SurfaceNet is a fully 3D convolutional network which is achieved by encoding the camera parameters together with the images in a 3D voxel representation. We evaluate SurfaceNet on the large-scale DTU benchmark.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “The problem of reconstructing 3D geometry from multiple input views has been considered in [31, 35, 52].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “We refer to the ﬁrst one as SurfaceNet (HD) and to the latter as SurfaceNet (LR) .”
TensoRF: Tensorial Radiance Fields (2022, background): “Various scene representations, including meshes [10,39], point clouds [31], volumes [13,32], implicit functions [22,30], have been extensively studied in recent years.”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, background): “…0.777 Tola[39] 0.342 1.190 0.766 Camp[3] 0.835 0.554 0.695 Gipuma[12] 0.283 0.873 0.578 Colmap[33, 34] 0.400 0.664 0.532 L ea r n i ng SurfaceNet[17] 0.450 1.040 0.745 MVSNet[42] 0.396 0.527 0.462 P-MVSNet[26] 0.406 0.434 0.420 R-MVSNet[43] 0.383 0.452 0.417 MVSCRF[40] 0.371 0.426 0.398…”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “[116] split the whole space into small Colored Voxel Cubes (CVCs) and regress the surface cube-by-cube.”
Deep Volumetric Video From Very Sparse Multi-view Performance Capture (2018, methodology): “As a learning-based approach, SurfaceNet [74] reconstructs a more complete point cloud, but still fails at the region with fewer correspondences due to large baseline.”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Different from depth estimation, 3D scene reconstruction methods [23, 37, 49, 6, 8] directly reconstruct a comprehensive and accurate 3D geometry of a scene.”
Point-Based Multi-View Stereo Network (2019, background): “Concurrently, 3D cost volume regularization approaches have been proposed [14, 12, 13], where a 3D cost volume is built either in the camera frustum or the scene.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, background): “Since convolutions are natural on voxels, they have been the most popular representation for learning [39, 35, 65], but the memory footprint scales cubically with resolution, which has limited grids to small sizes of 32 3 [44, 83, 16, 75].”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, methodology): “[28] also uses a similar approach, but requires at least two views.”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “An alternative way to combine several input views with 3D, is by projecting colors directly into the voxels, maintaining one channel for each input view per voxel [30].”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Voxels for rigid objects: Since voxels are a natural 3D extension to pixels in image grids and admit 3D convolutions, they are most commonly used for generation and reconstruction [29, 26, 57, 46].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, methodology): “Volumetric based approaches partition the space into a regular 3D volume with millions of small voxels [23, 27, 54, 55, 60, 40], and the network predicts if a voxel is on the surface or not.”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Recently, [40] [41] [42] [43] [44] [45] leverage deep neural nets to learn the 3D shape from multiple RGB images.”
MVSNet: Depth Inference for Unstructured Multi-view Stereo (2018, methodology): “Inspired by recent learning-based stereo [14] and MVS [17,18] methods, we apply the multi-scale 3D CNN for cost volume regularization.”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “Trained on DTU [2] As suggested by previous methods [30, 10, 31], DTU dataset is divided into training, validation and evaluation sets.”
Point-NeRF: Point-based Neural Radiance Fields (2022, background): “Traditional and neural methods have studied many 3D scene representations, including volumes [18, 24, 40, 44, 52], point clouds [1, 39, 48], meshes [19, 49], depth maps [16, 27], and implicit functions [9, 32, 36, 55], in diverse vision and graphics applications.”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “SurfaceNet [13] and DeepMVS [11] pre-warp the multi-view images to 3D space and use deep networks for regularization and aggregation.”
Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference (2019, methodology): “Heuristic divide-and-conquer strategies are applied in both classical [18] and learned MVS approaches [14, 13], however, usually lead to the loss of global context information and the slow processing speed.”