How Useful Is Photo-Realistic Rendering for Visual Learning?

By Yair Movshovitz-Attias, Takeo Kanade, Yaser Sheikh. 2016

158 citations recorded by the source corpus.

Data seems cheap to get, and in many ways it is, but the process of creating a high quality labeled dataset from a mass of data is time-consuming and expensive.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, methodology): “More recently, 3D models have been used to augment training data to aid transferring deep neural networks between datasets and prevent over-fitting on small datasets for tasks like viewpoint estimation [43] and object detection [45], [31].”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “[36,47] use rendered images of objects to do both object detection and pose estimation. They render 3D models of objects from different viewpoints and place them against randomly sampled backgrounds. [34] also highlight the importance of using photorealsitic models in training CNNs. There is a wide ”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “[17] used environment map lighting and showed that it benefits pose estimation.”
Modeling Visual Context is Key to Augmenting Object Detection Datasets (2018, background): “Regarding image synthesis, recent works such as [31,32,33] build and train their models on purely synthetic rendered 2d and 3d scenes.”
Implicit 3D Orientation Learning for 6D Object Detection from RGB Images (2018, background): “Photo-Realistic Rendering of object views and backgrounds has shown mixed generalization performance for tasks like object detection and viewpoint estimation [26,34,25,30].”
Adversarial Geometry and Lighting using a Differentiable Renderer (2018, background): “Many previous works use rendered images during training, as rendering provides a theoretically inﬁnite supply of input data [29, 5, 43, 39, 33, 16].”
Incremental Object Learning From Contiguous Views (2019, background): “Multiple synthetic optical ﬂow datsets [33, 5, 46] have led to performance improvements, as have generated 3D car assets from [35].”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, unknown): “Synthetic image datasets have recently been a source of training data for object detection and correspondence matching (Stark et al. 2010; Sun andSaenko2014;SongandXiao2014; Fanello et al. 2014; Dosovitskiy et al. 2015; Peng et al. 2015; Zhou et al. 2016; Gaidon et al. 2016;Movshovitz-Attias et al. ”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “In computer vision, synthetic-to-real (syn2real) transfer is a promising strategy that has been attracting attention [9, 12, 22, 29, 44, 56, 61].”
Synthetic Data for Deep Learning (2019, unknown): “[419].”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “Synthetic data has been successfully used in a range of application domains including prediction of object pose [33, 21, 11], optical flow [6], semantic segmentation for indoor scenes [12, 36], and analysis of image features [2, 15].”
Augmented Reality meets Deep Learning (2017, background): “A common strategy to circumvent the need for human labels is to train neural networks on synthetic data obtained from a 3D renderer for which ground truth labels can be automatically obtained [8, 11, 16, 20, 21, 23, 29, 31].”
A self-supervised learning system for object detection using physics simulation and multi-view pose estimation (2017, methodology): “Synthetic Datasets: Synthetic datasets generated from 3D models have been used for object detection [6], [26] and pose estimation [27], [7] with mixed success as indicated by an evaluation of the performance of detectors trained on synthetic images to those trained with natural images [9].”