ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
By Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, Matthias Nießner. 2017
5,692 citations recorded by the source corpus.
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.
Subfields: Aerospace Engineering, Environmental Engineering, Geology, Evaluation & Benchmarks, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Andreas Geiger
- Marc Pollefeys
- Vladlen Koltun
- Jitendra Malik
- Matthias Nießner
- Andrew J. Davison
- Silvio Savarese
- Leonidas J. Guibas
In Environmental Engineering
- Matthias Nießner
- Angela Dai
- Jitendra Malik
- Andreas Geiger
- Vladlen Koltun
- Silvio Savarese
- Armand Joulin
- Charles Ruizhongtai Qi
In Geology
- Matthias Nießner
- Leonidas J. Guibas
- Charles Ruizhongtai Qi
- Andreas Geiger
- Angela Dai
- Or Litany
- Jitendra Malik
- Vladlen Koltun
In Evaluation & Benchmarks
- Jitendra Malik
- Antonio Torralba
- Vladlen Koltun
- Luc Van Gool
- Trevor Darrell
- Kaiming He
- Abhinav Kumar Gupta
- Alexei A. Efros
In 3D Vision & Pose
- Andreas Geiger
- Jitendra Malik
- Matthias Nießner
- Angela Dai
- Michael Niemeyer
- Leonidas J. Guibas
- Justin Johnson
- Thomas A. Funkhouser
Citations in the corpus
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, neutral): “Most RGB-D datasets are derived from video sequences; e.g. NYUv2 [36], SUN RGB-D [40] and ScanNet [13].”
- ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Detection and Segmentation Datasets Many datasets for image segmentation have been proposed with the goal of densely recognizing general objects and “stuff” in image scenes like street view [7, 17, 61], natural scenes [9, 20, 35, 44,65], and indoor spaces [13,19,53].”
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “The prevalence of these classes is roughly consistent with existing indoor scene datasets [19, 69, 72].”
- PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017, methodology): “Semantic scene labeling is evaluated by average voxel classification accuracy following [5].”
- Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “RGB-D and 3D scene meshes datasets [19, 9, 1, 37, 34] have more objects that are reconstructed along with the scenes, but such objects are often considered in a scene segmentation or object detection task, and not under an object classification setup.”
- Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection (2021, background): “ScanNet (Dai et al. 2017) consists of 1,201 training samples and 312 validation samples, where there is no amodal oriented 3D bounding boxes but point-level semantic segmentation labels.”
- Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Notable attempts include LabelMe3D [46], NYU-D [50], SUN RGB-D [53], KITTI [16], and modern large-scale RGB-D scene datasets [10, 40, 54].”
- OctNetFusion: Learning Depth Fusion from Data (2017, background): “This has led to the creation of large databases of real-world 3D content [7, 10, 42, 43], enabling progress in a variety of areas including 3D modeling [28, 54], 3D reconstruction [6, 52], 3D recognition [18, 44] and 3D scene understanding [16, 42].”
- Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, methodology): “We synthesize from coarse to fine resolution, initializing with the coarse prediction P and computing a multi-scale representation of P 0 .”
- The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, methodology): “ScanNet [14] design an easy-to-use RGB-D capture system to collect 1,500 real-world scene scans, along with many semantic annotations.”
- Cognitive Mapping and Planning for Visual Navigation (2017, background): “Numerous works such as (Chang et al. 2017; Dai et al. 2017; Ammirato et al. 2017) have collected large-scale datasets consisting of scans of real world environments, while (Savva et al. 2017; Wu et al. 2018; Xia et al. 2018) have built more sophisticated simulation environments based on such scans.”
- Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “Previous RGB-D datasets have provided data for single rooms or small sets of adjacent rooms [44, 7], or single floors of a building [2].”
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, neutral): “they consist of RGB images with corresponding depth annotation of some form [3], [11], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [40], [44], [45], [46], [47], [48].”
- Spherical CNNs on Unstructured Grids (2019, background): “In the 3D learning literature, researchers have looked at 3D semantic segmentation on point clouds or voxels (Dai et al., 2017a; Qi et al., 2017a; Wang et al., 2018b; Tchapmi et al., 2017; Dai et al., 2017b).”
- The Replica Dataset: A Digital Replica of Indoor Spaces (2019, methodology): “Beyond Matterport-scanner-based reconstructions, there is the ScanNet [8] dataset which was obtained by scanning scenes with an iPad-based RGB-D camera system.”
- PointCNN: Convolution On X-Transformed Points (2018, methodology): “7 Table 1: Comparisons of mean per-class accuracy (mA) and overall accuracy (OA) (%) on ModelNet40 [52] and ScanNet [9].”
- MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, methodology): “Input images in this figure were taken (top to bottom) from the ScanNet [8], WildDash [44], and Pascal VOC [10] datasets, none of which were seen during training.”
- 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, methodology): “We evaluate our results with per-voxel class accuracies, following the evaluations of previous work [1,24,12].”
- Continuous 3D Perception Model with Persistent State (2025, positive): “Following MonST3R [98], we evaluate camera pose estimation accuracy on Sintel [10], TUM dynamics [72], and Scan-Net [15] datasets.”