Understanding RealWorld Indoor Scenes with Synthetic Data

By Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla. 2015

237 citations recorded by the source corpus.

Scene understanding is a prerequisite to many high level tasks for any automated intelligent machine operating in real world environments. Recent attempts with supervised learning have shown promise in this direction but also highlighted the need for enormous quantity of supervised data- performance increases in proportion to the amount of data used. However, this quickly becomes prohibitive when considering the manual labour needed to collect such data. In this work, we focus our attention on depth based semantic per-pixel labelling as a scene understanding problem and show the potential of computer graphics to generate virtually unlimited labelled data from synthetic 3D scenes. By carefully synthesizing training data with appropriate noise models we show comparable performance to state-of-the-art RGBD systems on NYUv2 dataset despite using only depth data as input and set a benchmark on depth-based segmentation on SUN RGB-D dataset.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “ods to generate realistic camera jitter [41]; and using semantic segmentation information to match the distribution of semantic classes to an existing dataset [25], prefer foreground semantic classes [27,28,65], and maintain a minimum number of semantic instances in each view [49,78]. However, exist”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “r from simulation to the real world has been studied extensively in computer vision and robotics. Synthetic data has been used for training and evaluation of perception systems in indoor environments [54,18,30] and driving scenarios [40,15, 29,38,45,22,47,2]. Direct transfer from simulation to reali”
Playing for Data: Ground Truth from Computer Games (2016, background): “formance of semantic segmentation systems. Somewhat orthogonal to our work is the use of indoor scene models to train deep networks for semantic understanding of indoor environments from depth images [15, 33]. These approaches compose synthetic indoor scenes from object models and synthesize depth m”
Joint 2D-3D-Semantic Data for Indoor Scene Understanding (2017, background): “Although several RGB-D [6, 7, 16, 15] and a few 3D datasets [12, 13, 14, 1] have been developed to date, the majority of them are limited in the scale, diversity, and/or the number of modalities they provide.”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “Computer games and simulated platforms have been used for training perceptual tasks, such as semantic segmentation [18], pedestrian detection [33], pose estimation [38], and urban driving [5, 41, 42, 45].”
Playing for Benchmarks (2017, methodology): “Virtual environments have also been used to collect training data for pedestrian detection [40], stereo reconstruction and optical ﬂow estimation [41], and semantic segmentation [23, 55].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “ss a breadth of computer vision problems, including optical ﬂow [148, 24, 156, 147], object trackers [225, 91], depth estimation from RGB [192], object detection [144, 234, 253] semantic segmentation [109, 179, 176] or human actions recognition [211]. In most cases, the synthetic data is used to enr”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, methodology): “Simulation has been used for a number of tasks in computer vision and robotics, including object recognition [11], semantic segmentation [16], robot grasping [12], and autonomous driving [17].”
A research agenda for augmented and virtual reality in architecture, engineering and construction (2020, background): “[72], (iii) leveraging transfer learning techniques by using datasets developed for other tasks [197], and (iv) investigating the applicability of unsupervised machine learning algorithms that do not require large labelled datasets for training [49].”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “Various works have proposed generating synthetic data for tasks such as perception and planning in driving [55,18,53,14,48,68,2], indoor scene perception [70,75,46,57,59,24,4], game playing [6,28], robotic control [61] [6,63,56], optical flow estimation [7,58], home robotics [49,34,20] amongst many ”
SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks (2017, methodology): “We focus on the semantic segmentation task of the SUN RGB-D dataset, using the 13 classes defined in [45].”
Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes (2017, methodology): “tic data for 3D human pose estimation. In ([6]), synthetic videos are used for human action recognition with deep networks. [32] present a synthetic dataset for indoor scene understanding. Similarly, [11] use synthetic data to train a depth-based pixelwise semantic segmentation method. In [31], a sy”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “Synthetic data has been successfully used in a range of application domains including prediction of object pose [33, 21, 11], optical flow [6], semantic segmentation for indoor scenes [12, 36], and analysis of image features [2, 15].”
Local Implicit Grid Representations for 3D Scenes (2020, unknown): “Data We evaluate the reconstruction performance of the methods on a synthetic dataset: SceneNet [16], and a”