SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth

By John McCormac, Ankur Handa, Stefan Leutenegger, Andrew J. Davison. 2016

135 citations recorded by the source corpus.

We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems such as semantic segmentation, instance segmentation, and object detection, and also for geometric computer vision problems such as optical flow, depth estimation, camera pose estimation, and 3D reconstruction. Random sampling permits virtually unlimited scene configurations, and here we provide a set of 5M rendered RGB-D images from over 15K trajectories in synthetic layouts with random but physically simulated object poses. Each layout also has random lighting, camera trajectories, and textures. The scale of this dataset is well suited for pre-training data-driven computer vision techniques from scratch with RGB-D inputs, which previously has been limited by relatively small labelled datasets in NYUv2 and SUN RGB-D. It also provides a basis for investigating 3D scene labelling tasks by providing perfect camera poses and depth data as proxy for a SLAM system. We host the dataset at this http URL

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “SceneNet RGB-D [9], [10] and AI2-THOR [1], [11] are two recently proposed synthetic 3D scenes datasets.”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “HoME [5], House3D [2] and MINOS [17] (which are based on SUNCG [18] and/or Matterport3D [6]) and SceneNet RGBD [15] also provide synthetic indoor environments.”
Joint 2D-3D-Semantic Data for Indoor Scene Understanding (2017, background): “Although several RGB-D [6, 7, 16, 15] and a few 3D datasets [12, 13, 14, 1] have been developed to date, the majority of them are limited in the scale, diversity, and/or the number of modalities they provide.”
Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes (2017, methodology): “By matching the spatial distributions of object categories in example images, we may be able to train deep networks more effectively than is possible now with cameras placed heuristically [40], along trajectories [23], or randomly [13].”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “Various works have proposed generating synthetic data for tasks such as perception and planning in driving [55,18,53,14,48,68,2], indoor scene perception [70,75,46,57,59,24,4], game playing [6,28], robotic control [61] [6,63,56], optical flow estimation [7,58], home robotics [49,34,20] amongst many ”
Meaningful maps with object-oriented semantic mapping (2016, unknown): “The proposed future research can be supported by recently published synthetic datasets [38] or high-fidelity”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, background): “Applications like optical flow [1], [3], scene flow [4], classification [18], stereo [5], [6], semantic segmentation [8], [9], [19], 3D keypoint extraction [20], object pose [11] and 3D reconstruction [2], [7] have all benefited from the use of synthetic training data.”
ViewAL: Active Learning With Viewpoint Entropy for Semantic Segmentation (2019, methodology): “Figure 1: ViewAL is an active learning method that significantly reduces labeling effort: with maximum performance attained by using 100% of the data (last column), ViewAL achieves 95% of this performance with only 7% of data of SceneNetRGBD [29].”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “The community has been investing significant effort in creating high-quality synthetic content, ranging from driving scenes [38, 11, 36, 8, 34, 47, 2], indoor scenes [48, 50, 33], household robotics [35, 22], robotic control [44], game playing [5], optical flow estimation [6, 23], and quadcopter con”
InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset (2018, methodology): “2 (D)) to assist creating monocular or stereo camera trajectories, ground truth IMU readings and events given a layout from our database or from any other 3D scenes with obj format e.g. SUNCG [28] and SceneNet RGB-D [17] etc.”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, background): “Given the deep learning hunger for large amounts of labeled training data, a recent research trend has focused on providing synthetic datasets for training [38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 20, 48].”
Are We Ready for Service Robots? The OpenLORIS-Scene Datasets for Lifelong SLAM (2019, background): “Recent progress in random scene generation and photo-realistic rendering [6][7] makes it theoretically possible to synthesize scene changes for lifelong SLAM, but it would be difﬁcult to model realistic changes as in natural lives.”
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (2018, background): “The use of synthetic data for training and testing deep neural networks has gained in popularity in recent years, as evidenced by the availability of a large number of such datasets: Flying Chairs [4], FlyingThings3D [18], MPI Sintel [1], UnrealStereo [24, 36], SceneNet [9], SceneNet RGB-D [19], SYN”
Dropout Sampling for Robust Object Detection in Open-Set Conditions (2017, methodology): “Our evaluation is based on two datasets: SceneNet RGBD [26], a huge dataset of rendered scenes, and a smaller real- world dataset captured by our robot in a variety of indoor and outdoor environments on our campus [27]. a) SceneNet RGB-D: The SceneNet RGB-D validation set contains photo-realistic im”
Street-view change detection with deconvolutional networks (2016, background): “2016) and indoor scene understanding (Handa et al. 2016b, a;McCormac et al. 2016; Zhang et al. 2016).”
Image Inpainting: A Review (2019, unknown): “SceneNet dataset10 [102] is a dataset for scene understanding tasks including semantic segmentation, object detection and 3D reconstruction.”