Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

By Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, Thomas A. Funkhouser. 2016

270 citations recorded by the source corpus.

Indoor scene understanding is central to applications such as robot navigation and human companion assistance. Over the last years, data-driven deep neural networks have outperformed many traditional approaches thanks to their representation learning capabilities. One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object boundary detection. To address this problem, a number of works proposed using synthetic data. However, a systematic study of how such synthetic data is generated is missing. In this work, we introduce a large-scale synthetic dataset with 500K physically-based rendered images from 45K realistic 3D indoor scenes. We study the effects of rendering methods and scene lighting on training for three computer vision tasks: surface normal prediction, semantic segmentation, and object boundary detection. This study provides insights into the best practices for training with synthetic data (more realistic rendering is worth it) and shows that pretraining with our new synthetic dataset can improve results beyond the current state of the art on all three tasks.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “…the color image. alistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [6, 7, 9, 10, 14, 15, 19, 22, 24, 31, 33, 36, 37, 39, 43, 44, 45, 46, 49, 52, 61, 62, 63, 65, 70, 72, 75, 76, 79, 83, 84].”
IRS: A Large Synthetic Indoor Robotics Stereo Dataset for Disparity and Surface Normal Estimation (2019, background): “Recent work in [14], [15], [16] leveraged synthetic technology to generate sufficiently large data volume for DNN training.”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “Existing 3D scenes datasets supporting the training of robot navigation can generally be categorized into two categories: 1) synthetic scenes [6], [7], [8], [9], [10], [11] 3D reconstructed scenes [12], [13], [14].”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “[48] demonstrate that by pretraining on a huge repository of high-quality synthetic data rendered from SUNCG [38] and then ﬁne-tuning on NYUv2, the network can achieve signiﬁcantly better performance than directly training on NYUv2.”
DeepMVS: Learning Multi-view Stereopsis (2018, background): “Examples of synthetic datasets for training and evaluating computer vision algorithms include indoor scene understanding [43], semantic segmentation [31, 30, 29], and depth and ﬂow estimation [27, 29].”
Synthetic Data for Deep Learning (2019, background): “[701] studied the importance of synthetic data realism for various indoor vision tasks.”
SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? (2017, methodology): “NYUv2 [29] SUN RGB-D [31] sceneNN [16] 2D-3D-S [1] ScanNet [7] SceneNet [13] SUN CG✄ [32, 33] SceneNet RGB-D”
3D-FUTURE: 3D Furniture Shape with TextURE (2020, background): “Lots of 3D benchmarks have been established andmade publicly available over the past decades (Chang et al. 2015; Wu et al. 2015; Xiao et al. 2013, 2016; Song et al. 2015; Sun et al. 2018a; Xiang et al. 2014, 2016; Zhang et al. 2017; Silberman et al. 2012; Dai et al. 2017a; Hua et al. 2016; Choi et a”
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “Figure 11 shows qualitative comparisons with [38] and [51] on three real examples from [38].”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, unknown): “It also aids in separating viewpoint effects in learning view-invariant representations like patch descriptors [40] and normals [41].”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “However, prior datasets usually only provide diffuse shading as their lighting representation [34, 61].”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “We use cosine loss following [56] for Ln.”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, background): “However, their materials are not realistic [55], so we replace them with photorealistic SVBRDFs from a high-quality 3D material dataset [50].”
Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene (2017, methodology): “[35] for our experiments and randomly partition the houses into a 70%-10%-20% train, validation and test split.”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “The community has been investing significant effort in creating high-quality synthetic content, ranging from driving scenes [38, 11, 36, 8, 34, 47, 2], indoor scenes [48, 50, 33], household robotics [35, 22], robotic control [44], game playing [5], optical flow estimation [6, 23], and quadcopter con”
The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions (2018, unknown): “Inst. 3D Object Pose Camera Pose Hand Pose Depth NYU-D-V2 [3] • • ∼1 • Sun RGB-D [4] • • • Stanford 2D-3D-S [5] • • • • • • Matterport 3D [6] • • • • • • Sun CG [7] • • PBR-Princeton [8] • • • SceneNet RGB-D [9] • • • • • Ours • • • • • • • • (Multi) • • HoMe [10] • • House3D [11] • • AI2-THOR [12] ”