ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

By Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott E. Reed, Jürgen Sturm, Matthias Nießner. 2017

332 citations recorded by the source corpus.

We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels. The key contribution of our method is its ability to handle large scenes with varying spatial extent, managing the cubic growth in data size as scene size increases. To this end, we devise a fully-convolutional generative 3D CNN model whose filter kernels are invariant to the overall scene size. The model can be trained on scene subvolumes but deployed on arbitrarily large scenes at test time. In addition, we propose a coarse-to-fine inference strategy in order to produce high-resolution output while also leveraging large input context sizes. In an extensive series of experiments, we carefully evaluate different model design choices, considering both deterministic and probabilistic models for completion and semantic inference. Our results show that we outperform other methods not only in the size of the environments handled and processing efficiency, but also with regard to completion quality and semantic segmentation performance by a significant margin.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Spherical CNNs on Unstructured Grids (2019, background): “In the 3D learning literature, researchers have looked at 3D semantic segmentation on point clouds or voxels (Dai et al., 2017a; Qi et al., 2017a; Wang et al., 2018b; Tchapmi et al., 2017; Dai et al., 2017b).”
TopNet: Structural Point Cloud Decoder (2019, background): “The ability to infer the full shape of an object from an incomplete scan acquired by depth camera or LiDAR is indeed an important task which enables several downstream applications such as robotics manipulation [31], scene understanding for navigation [7], and virtual manipulation of completed shape”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “ScanComplete [15] builds upon this and shows results for entire rooms.”
Fusion++: Volumetric Object-Level SLAM (2018, methodology): “A learned mechanism for filtering and reconstructing these objects, such as [7] may prove useful in this regard, or combining view-based segmentation and classification with 3D methods which take advantage of object databases such as ShapeNet [2].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, methodology): “Several works have followed, using 3D CNNs for object classification [14,10] or generative scene completion tasks [15,16,12].”
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (2018, methodology): “Trajectories are generated following the virtual scanning approach of [9], but adapted to provide denser camera trajectories to better simulate real-world scanning scenarios.”
Local Implicit Grid Representations for 3D Scenes (2020, unknown): “[8] uses a 3D convolutional network with a coarse-tofine inference strategy to directly regress gridded Truncated Signed Distance Function (TSDF) outputs from incomplete input TSDF.”
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, unknown): “Voxel-based Methods Early work [59, 10, 38] represented 3D shapes as voxels, which store the coarse occupancy (inside/outside) values on a regular grid, which makes powerful convolutional neural networks native and renders impressive results on 3D reconstruction and synthesis [12, 11, 58, 2].”
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (2021, methodology): “These include classic techniques that fuse observed depths into a single 3D reconstruction, typically in the form of a truncated signed distance function [5, 15], as well as more recent techniques that train deep networks to operate over the sparsely observed geometry in 3D [7,8].”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, methodology): “For indoor environments, there are several datasets [48, 46, 24, 3, 11, 35, 32, 12] available, which are mainly recorded using RGB-D cameras or synthetically generated.”
Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, methodology): “Mean-while, coarse-to-ﬁne strategies (e.g., LSTM based model) are used in (Dai et al. 2018; Han et al. 2017) to recover missing parts of 3D shapes.”
JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields (2019, background): “ed to limit convolution operations only on free-space voxels [35]. It has been shown that the performance of semantic segmentation can be improved by solving the problem jointly with scene completion [39,6]. Pointcloudapproach. In contrast to volume, point cloud is a compact yet intuitive representa”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, methodology): “[236] perform scene prediction over 3-D SDF-based maps.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “Although we have noticeable gap to the state-of-the-art fully-supervised approaches, our zero-shot method is surprisingly competitive with fullysupervised approaches from a few years ago [12, 24, 51].”
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans (2018, background): “Learning-based approaches for object and scene completion [9, 37, 10] cannot reliably recover sharp edges or planar surfaces, resulting in quality far from artist-modeled 3D content.”
Shape As Points: A Differentiable Poisson Solver (2021, background): “Implicit representations, in contrast, represent the shape as a level set of a continuous function over a discretized voxel grid [14, 27, 35, 69] or more recently parameterized as a neural network, typically referred to as neural implicit functions [10,42,52].”
Unsupervised 3D Shape Completion through GAN Inversion (2021, background): “3D shape completion has played an important role for robotics [11, 24] and perception [9, 14].”