Discrete optimization of ray potentials for semantic 3D reconstruction

By Nikolay Savinov, Lubor Ladicky, Christian Häne, Marc Pollefeys. 2015

65 citations recorded by the source corpus.

Dense semantic 3D reconstruction is typically formulated as a discrete or continuous problem over label assignments in a voxel grid, combining semantic and depth likelihoods in a Markov Random Field framework. The depth and semantic information is incorporated as a unary potential, smoothed by a pairwise regularizer. However, modelling likelihoods as a unary potential does not model the problem correctly leading to various undesirable visibility artifacts. We propose to formulate an optimization problem that directly optimizes the reprojection error of the 3D model with respect to the image estimates, which corresponds to the optimization over rays, where the cost function depends on the semantic class and depth of the first occupied voxel along the ray. The 2-label formulation is made feasible by transforming it into a graph-representable form under QPBO relaxation, solvable using graph cut. The multi-label problem is solved by applying α-expansion using the same relaxation in each expansion move. Our method was indeed shown to be feasible in practice, running comparably fast to the competing methods, while not suffering from ray potential approximation artifacts.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “Occlusion constraints are encoded using high-order ray potentials [19, 28, 35].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, neutral): “depth or semantics [20, 29, 30].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “combine ray potentials with 3D shape regularizers that are learned from data [38].”
Weakly Supervised 3D Reconstruction with Adversarial Constraint (2017, methodology): “Among many lines of work within the 3D reconstruction [18, 25, 13, 14, 3, 8, 23, 38, 49, 35], ours is related to recent works that use neural networks for 3D voxel reconstruction.”
Semantic Multi-view Stereo: Jointly Estimating Objects and Voxels (2017, methodology): “exploit scene semantics and propose class-specific pairwise priors [30].”
From Point Clouds to Mesh Using Regression (2017, methodology): “Using recent advances in optimization Savinov et al. [23] formulate the visibility constraints as high order potentials over rays and solves it using non-convex relaxation methods.”
Large-Scale Semantic 3D Reconstruction: An Adaptive Multi-resolution Model for Multi-class Volumetric Labeling (2016, background): “Jointly inferring 3D geometry and semantics is a hard problem, and has only recently been tackled in a principled manner [17, 25, 36].”
Towards Probabilistic Volumetric Reconstruction Using Ray Potentials (2015, methodology): “While previous works on ray potentials aim at the maximum-a-posteriori (MAP) solution of the occupancy variables [20, 28], our goal is to infer the marginal distributions of occupancy and appearance at each voxel.”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “Typically, these approaches operate on a single RGB image; however, one can easily formulate the analogous task in 3D on a per-voxel basis [5, 13, 21, 34, 40, 41], which is a common scenario in the context of 3D scene reconstruction.”
Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint (2016, methodology): “This problem has been addressed in [26], but their discrete graph-based approach suffers from metrication artifacts, cannot be combined with the class-specific anisotropic smoothness prior and does not lead to smooth surfaces (c.”
Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction (2016, background): “Previous works impose smoothness constraints via pairwise potentials that encourage adjacent voxels to take on the same occupancy state [18, 31] or condition surface orientations on semantic information [40].”
Dense Semantic 3D Reconstruction (2017, methodology): “Follow up works of our method extend our method to work with a ray potential data term [32], [33] and on octrees [2].”