RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

By Despoina Paschalidou, Ali O. Ulusoy, Carolin Schmitt, Luc Van Gool, Andreas Geiger. 2018

98 citations recorded by the source corpus.

In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Differentiation of Blackbox Combinatorial Solvers (2019, methodology): “The techniques used for constructing the backward pass range from employing various relaxations and approximations of the combinatorial problem [8, 53] over differentiating a fixed number of iterations of an iterative solver [28, 34, 48] all the way to relying on the structured SVM framework [8, 49]”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “LSM [11] and RayNet [21] encoded the camera projection to the network, and utilized 3D CNNs or Markov Random Field to predict surface label.”
Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction (2018, methodology): “, 2018) simply use max pooling to preserve the first order information of multiple images, while RayNet (Paschalidou et al., 2018) applies average pooling to reserve the first moment information of multiple deep features.”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, methodology): “To overcome such limitations, many deep learning-based MVS methods have emerged in the last two years, including regression-based approaches [58, 21], classification-based approaches [20] and approaches based on recurrent- or iterative- style architectures [59, 61, 5] and many other approaches [30, ”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “Recently, learning-based approaches [16, 29, 39, 58, 63, 86, 87] have been proposed that either learn to match image features [39], refine or fuse depth maps [16, 63], optimize parts of the classical MVS pipeline [57], or replace the entire MVS pipeline with neural networks that are trained end-to-e”
Multi-view 3D Reconstruction with Transformers (2021, background): “The poolingbased fusion only learns partial information of multiple views and ignores the view associations [11, 15].”
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction (2021, background): “Before the era of deep learning, classic multi-view stereo (MVS) methods [2, 4, 5, 7, 21, 21, 48, 50, 51] focus on either matching features across views [4, 48] or representing shapes with a voxel grid [2, 5, 7, 21, 28, 42, 50, 54, 55].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “Classic multi-view stereo (MVS) methods [2, 6–8, 35, 35, 62, 64, 65] consider either feature matching for depth estimation [6, 62] or represent shapes with voxels [2, 7, 8, 35, 51, 64, 72, 73].”
Neural volumes (2019, background): “[2016], and Paschalidou et al. [2018] formulate graph-based energy or inference objectives using ray potentials as constraints.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “The approach of Paschalidou et al. [91] takes approximately 25 mins, on an Intel i7 computer with an Nvidia GTX Titan X GPU, for the same task.”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, methodology): “[25] unrolls the MRF optimization to train it jointly with a network for depth regression.”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, background): “More recently, a few approaches have proposed to directly solve MVS problems [14, 24, 29].”