Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

By Nikolay Savinov, Christian Häne, Lubor Ladicky, Marc Pollefeys. 2016

65 citations recorded by the source corpus.

We propose an approach for dense semantic 3D reconstruction which uses a data term that is defined as potentials over viewing rays, combined with continuous surface area penalization. Our formulation is a convex relaxation which we augment with a crucial non-convex constraint that ensures exact handling of visibility. To tackle the non-convex minimization problem, we propose a majorizeminimize type strategy which converges to a critical point. We demonstrate the benefits of using the non-convex constraint experimentally. For the geometry-only case, we set a new state of the art on two datasets of the commonly used Middlebury multi-view stereo benchmark. Moreover, our general-purpose formulation directly reconstructs thin objects, which are usually treated with specialized algorithms. A qualitative evaluation on the dense semantic 3D reconstruction task shows that we improve significantly over previous methods.

Subfields: Aerospace Engineering, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning a Multi-View Stereo Machine (2017, methodology): “Learning has been used for multi-view reconstruction in the form of shape priors for objects [2, 9, 58, 20, 27, 52], or semantic class speciﬁc surface priors for scenes [22, 17, 45].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, neutral): “depth or semantics [20, 29, 30].”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “These types of supervisions utilize ray formulations which originally have been proposed for multiview reconstruction [23], [29].”
Dense Semantic 3D Reconstruction (2017, methodology): “Follow up works of our method extend our method to work with a ray potential data term [32], [33] and on octrees [2].”
Multi-Label Semantic 3D Reconstruction Using Voxel Blocks (2016, background): “Using a data term defined as ray potential could further improve the quality of the reconstructions [22].”
Neural RGB-D Surface Reconstruction (2021, background): “In addition, several regularization techniques have been proposed to handle outliers during reconstruction [19, 61, 88].”