Joint 2D-3D-Semantic Data for Indoor Scene Understanding

By Iro Armeni, S. Sax, Amir Zamir, Silvio Savarese. 2017

1,038 citations recorded by the source corpus.

We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. The dataset covers over 6,000m2 and contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360{\deg} equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. The dataset is available here: this http URL

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Environmental Engineering, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Geology

In 3D Vision & Pose

Citations in the corpus

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “Refer to Stanford 2D-3D-S [5] for a recent example of an academic dataset collected with a Matterport camera.”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “…the color image. alistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [6, 7, 9, 10, 14, 15, 19, 22, 24, 31, 33, 36, 37, 39, 43, 44, 45, 46, 49, 52, 61, 62, 63, 65, 70, 72, 75, 76, 79, 83, 84].”
Embodied Question Answering (2017, neutral): “AI2-THOR [18], Matterport3D [31], Stanford 2D-3D-S [35]).”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, methodology): “For our ﬁnal experiment, we evaluate icosahedral CNNs on the 2D-3D-S dataset (Armeni et al., 2017), which consists of 1413 omnidirectional RGB+D images with pixelwise semantic labels in 13 classes.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “The methodologies and tools are scarce for collecting largescale world-based 3D scene datasets and 3D object assets [121]–[123].”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, methodology): “A large number of RGB-D datasets have been captured and made publicly available for training and benchmarking [56, 34, 50, 65, 79, 83, 74, 4, 58, 81, 15, 55, 1, 68, 30, 51, 21, 48, 43, 92, 80, 61, 72, 93, 36, 16, 35, 57, 40, 29, 70, 52, 45, 95, 75, 9, 33, 85, 71, 32, 3, 10, 78, 2].”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, methodology): “The Stanford 2D-3D-S [13], ScanNet [14] and Matterport3D [12] datasets provide 3D real-world scenes with raw 3D point clouds and reconstructed meshes.”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “The images are registered on and aligned with buildingwide meshes similar to [3, 101, 14] enabling us to programmatically compute the ground truth for many tasks without human labeling.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “The work most closely related to ours is by Armeni et al. [1].”
Gibson Env: Real-World Perception for Embodied Agents (2018, methodology): “The base format of the data is similar to 2D-3D-Semantics dataset [9], but is more diverse and includes 2 orders of magnitude more spaces.”
Spherical CNNs on Unstructured Grids (2019, methodology): “Armeni et al. (2017) and Chang et al. (2017) provide benchmarks for semantic segmentation of 360 panorama images.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “Stanford 2D-3D [170] provides a variety of mutually registered 2D, 2.”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “STANFORD 2D-3D-S [60] Indoor 70,496 Data is semantically annotated on 3D point clouds, rather than in 2D; the dataset includes only 13 object classes, which is few for an indoor environment.”
SO-Net: Self-Organizing Network for Point Cloud Analysis (2018, background): “Recent availability of 3D datasets including ModelNet [37], ShapeNet [8], 2D-3D-S [2] adds on to the popularity of research on 3D data.”
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI (2021, background): “Other datasets that provide 3D reconstructions of entire buildings such as the BuildingParser [3], Matterport3D [4] and Gibson [5] efforts are either limited in total size or suffer from incomplete reconstructions.”
Audio-Visual Embodied Navigation (2019, background): “Their realistic meshes can be rendered from agent-selected viewpoints to train and test RL policies for navigation in a reproducible manner [4, 13, 108, 56, 8, 85, 11, 107, 61].”
DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, background): “(3) We show how these new concepts help build a 56-layer GCN, the deepest GCN architecture by a large margin, and achieve close to 4% boost in state-of-the-art performance on the S3DIS dataset [1].”
Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery (2020, methodology): “We compare our method with other state-of-the-art approaches on the Stanford 2D-3D-S [1] dataset.”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, background): “Visual navigation has seen substantial progress in the past few years, fueled by large-scale datasets and photo-realistic 3D environments [4,9,74,69], simulators [74,34,3,38], and public benchmarks [13,3,38].”