Semantic Scene Completion from a Single Depth Image

By Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, Thomas A. Funkhouser. 2016

1,426 citations recorded by the source corpus.

This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created largescale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task. The dataset and code is available at http://sscnet.cs.princeton.edu.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For scenes, techniques include plane fitting [39], 3D model fitting and retrieval [15, 40] or leverage symmetries [28] and predict 3D semantics [4, 14, 59].”
Embodied Question Answering (2017, methodology): “We instantiate EmbodiedQA in House3D [1], a recently introduced rich, simulated environment based on 3D indoor scenes from the SUNCG dataset [8].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “VRKitchen consists of 16 fully interactive kitchen scenes, where the 3D models of furniture and appliances were imported from the SUNCG dataset [32].”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…depth sensors get mature [Izadi et al., 2011], and larger-scale RGB-D datasets become available [McCormac et al., 2017, Silberman et al., 2012, Song et al., 2017], many papers start to estimate depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang…”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “With the development of depth sensors [23] and larger-scale RGB-D datasets [33,42,43], there have also been papers on estimating depth [6,10], surface normals [1,54], and other intrinsic images [25,41] with deep networks.”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “Recent works show such a joint prediction improves over reconstruction or segmentation in isolation [28, 30].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “With the development of depth sensors [Izadi et al., 2011] and larger scale RGB-D datasets [Silberman et al., 2012, Song et al., 2017, McCormac et al., 2017], there have also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang…”
OctNetFusion: Learning Depth Fusion from Data (2017, methodology): “By utilizing large datasets of 3D models and high capacity 3D convolutional neural networks (3D CNNs), our approach learns to smooth out sensor noise, deals with outliers and completes missing 3D geometry.”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “Existing 3D scenes datasets supporting the training of robot navigation can generally be categorized into two categories: 1) synthetic scenes [6], [7], [8], [9], [10], [11] 3D reconstructed scenes [12], [13], [14].”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, methodology): “For indoor navigation simulators like AI2-THOR [29], SUNCG [30] and Matterport3D [31] have emerged.”
Unsupervised Learning of Disentangled Representations from Video (2017, methodology): “SUNCG: We use the rendering engine from the SUNCG dataset [30] to generate sequences where the camera rotates around a range of 3D chair models.”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “HoME [5], House3D [2] and MINOS [17] (which are based on SUNCG [18] and/or Matterport3D [6]) and SceneNet RGBD [15] also provide synthetic indoor environments.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “[48] demonstrate that by pretraining on a huge repository of high-quality synthetic data rendered from SUNCG [38] and then ﬁne-tuning on NYUv2, the network can achieve signiﬁcantly better performance than directly training on NYUv2.”
Joint 2D-3D-Semantic Data for Indoor Scene Understanding (2017, background): “Although several RGB-D [6, 7, 16, 15] and a few 3D datasets [12, 13, 14, 1] have been developed to date, the majority of them are limited in the scale, diversity, and/or the number of modalities they provide.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “House3D leverages the SUNCG dataset (Song et al., 2017) which contains 45K human-designed real-world 3D house models, ranging from single studios to houses with gardens, in which objects are fully labeled with categories.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “In the past two years, a plethora of tasks and evaluation protocols [32, 22, 34, 45, 2] have been proposed as summarized in [1].”