OctNetFusion: Learning Depth Fusion from Data

By Gernot Riegler, Ali O. Ulusoy, Horst Bischof, Andreas Geiger. 2017

211 citations recorded by the source corpus.

In this paper, we present a learning based approach to depth fusion, i.e., dense 3D reconstruction from multiple depth images. The most common approach to depth fusion is based on averaging truncated signed distance functions, which was originally proposed by Curless and Levoy in 1996. While this method is simple and provides great results, it is not able to reconstruct (partially) occluded surfaces and requires a large number frames to filter out sensor noise and outliers. Motivated by the availability of large 3D model repositories and recent advances in deep learning, we present a novel 3D CNN architecture that learns to predict an implicit surface representation from the input depth maps. Our learning based method significantly outperforms the traditional volumetric fusion approach in terms of noise reduction and outlier suppression. By learning the structure of real world 3D objects and scenes, our approach is further able to reconstruct occluded regions and to fill in gaps in the reconstruction. We demonstrate that our learning based approach outperforms both vanilla TSDF fusion as well as TV-L1 fusion on the task of volumetric fusion. Further, we demonstrate state-of-the-art 3D shape completion results.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “While most of these approaches encode objects in voxels from vision, there have also been attempts to reconstruct objects in point clouds [11,15] or octave trees [40,49,39], or using tactile signals [53].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “ge and shape together for single view 3D volumetric shape generation. Hierarchical space partitioning structures (e.g., octrees) have been proposed to increase the output resolution beyond 323 voxels [12,27,31]. As most of the aforementioned methods solve the 3D reconstruction problem via recognizin”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “ representation is costly in term of memory and is typically limited to coarser resolutions. To overcome this, recent work has looked at a voxel representation of the surface of a shape via oct-trees [13, 27, 31]. Recently, Li et al. also attempted to address this issue via learning to reason over h”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 42, 32, 12, 39, 34], such as decomposing image silhouettes [23? ], Light Field Descriptors [2], and 2D panoramic mapping [38].”
Learning a Multi-View Stereo Machine (2017, background): “We can borrow ideas from very recent works [43, 21] which show that CNNs can predict such high resolution volumes.”
Local Deep Implicit Functions for 3D Shape (2019, methodology): “Learned Shape Representations: To leverage shape priors, shape reconstruction methods began representing shape as a learned feature vector, with a trained decoder to a mesh [36, 14, 41, 15, 18], point cloud [10, 21, 46], voxel grid [9, 43, 5, 42], or octree [37, 33, 32].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “Classic approaches usually fuse multiple depth images through iterative closest point (ICP) algorithms [3] [46] [47], while recent work [48] learns the 3D shape using deep neural nets from multiple depth views.”
TopNet: Structural Point Cloud Decoder (2019, methodology): “This issue has been addressed using sparse representations [30, 32, 33, 17, 12, 25] which makes processing voxels more efficient.”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “Recently, learning-based approaches [16, 29, 39, 58, 63, 86, 87] have been proposed that either learn to match image features [39], refine or fuse depth maps [16, 63], optimize parts of the classical MVS pipeline [57], or replace the entire MVS pipeline with neural networks that are trained end-to-e”
Convolutional Occupancy Networks (2020, background): “Due to the cubic memory requirements of voxel-based representations, several works proposed to operate on multiple scales or use octrees for efficient space partitioning [8, 14, 25, 37, 38, 42].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “In order to address the memory and compute requirements, hierarchical 3D CNNs have been proposed to more eﬃciently represent and process 3D volumes [17,18,19,20,21,16].”
BSP-Net: Generating Compact Meshes via Binary Space Partitioning (2019, background): “To generate finer results, differentiable marching cubes operations have been proposed [26], as well as hierarchical strategies [18, 36, 42, 45, 46] that alleviate the curse of dimensionality affecting dense volumetric grids.”
Tangent Convolutions for Dense Prediction in 3D (2018, methodology): “heir prediction accuracy. The problem can be alleviated by octree-based techniques that deﬁne a ConvNet on an octree and enable processing somewhat higher-resolution volumes (e.g., up to 2563 voxels) [43,57,18,42,51]. Yet even this may be insufﬁcient for detailed analysis of large-scale scenes. On a”
Implicit Geometric Regularization for Learning Shapes (2020, background): “More generally, researchers have suggested us-ing other RGVFs (Dai et al., 2017; Riegler et al., 2017; Stutz & Geiger, 2018; Liao et al., 2018; Michalkiewicz et al., 2019; Jiang et al., 2019).”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “Octree-based decoders [Häne et al. 2017; Meagher 1980; Riegler et al. 2017b,a; Tatarchenko et al. 2017; Wang et al. 2017, 2018] and sparse hash-based decoders [Dai et al. 2020] take 3D space sparsity into account, alleviating the efficiency issues and supporting high-resolution outputs.”
3D Object Classification via Spherical Projections (2017, background): “Although this issue has been recently alleviated by Octreebased representations [19, 20, 26], the cost of 3D volumetric neural networks is still significant higher than 2D neural networks of the same resolution.”