Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization

By Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, Stan Birchfield. 2018

1,009 citations recorded by the source corpus.

We present a system for training deep neural networks for object detection using synthetic images. To handle the variability in real-world data, the system relies upon the technique of domain randomization, in which the parameters of the simulator-such as lighting, pose, object textures, etc.-are randomized in non-realistic ways to force the neural network to learn the essential features of the object of interest. We explore the importance of these parameters, showing that it is possible to produce a network with compelling performance using only non-artistically-generated synthetic data. With additional fine-tuning on real data, the network yields better performance than using real data alone. This result opens up the possibility of using inexpensive synthetic data for training neural networks while avoiding the need to collect large amounts of hand-annotated real-world data or to generate high-fidelity synthetic worlds-both of which remain bottlenecks for many applications. The approach is evaluated on bounding box detection of cars on the KITTI dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-Agent Common Knowledge Reinforcement Learning (2018, methodology): “We are also interested in utilising SIM2Real transfer methods (Tobin et al., 2017; Tremblay et al., 2018) in order to apply MACKRL to autonomous car and unmanned aerial vehicle coordination problems in the real world.”
Meta-Learning in Neural Networks: A Survey (2020, background): “tion approach simulates a wide distribution over domains/ MDPs, with the aim of training a sufficiently robust model to succeed in the real world – and has succeeded in both vision [272] and RL [158].”
Deep Template-based Object Instance Detection (2021, methodology): “Inspired from the success of domain randomization [36, 37], we added randomness to the simulation parameters in order to reduce the domain gap between synthetic and real images.”
Synthetic Data for Deep Learning (2019, methodology): “[595], who trained object detection models on synthetic data with the following procedure:”
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020, background): “In robotic vision tasks including object localization [53], object detection [54], pose estimation [55], and semantic segmentation [56], the training data from simulator always have different textures, lighting, and camera positions from the realistic environments.”
Geometric Image Synthesis (2018, background): “Yang et al. [40] introduce an additional diversity loss to generate more diverse outputs.”
Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization Without Accessing Target Domain Data (2019, methodology): “DR has also been utilized to do object detection and 6D pose estimation [49, 36, 45].”
Multiscale modeling meets machine learning: What can we learn? (2019, methodology): “Recent studies have shown that, in the area of object detection in image analysis, simulation augmented by domain randomization can be used successfully as a supplement to existing training data [131].”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “In computer vision, synthetic-to-real (syn2real) transfer is a promising strategy that has been attracting attention [9, 12, 22, 29, 44, 56, 61].”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “Similar techniques were also used in [29], [33], [34], [35] to strengthen the generalization capability”
FSDR: Frequency Space Domain Randomization for Domain Generalization (2021, background): “Domain Randomization (DR) is the common strategy in domain generalization [64, 53, 49, 63, 65, 59, 70, 77, 50] especially when a single source domain is available [64, 70, 77, 50].”
Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences (2019, methodology): “Example: Supplementing trianing data.(83) Dropout neural networks are a regularization method for neural networks that avoids overfitting by randomly deleting, or dropping, units along with their connections during training.”
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (2018, background): “Further improvements may be achieved by leveraging recent work in domain adaptation [24, 54, 50, 5, 58].”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, methodology): “For example, synthetic rendered images are used in stereo matching [3, 19, 36], optical ﬂow [3, 19, 5], object detection [6, 31] and semantic segmentation [6, 22, 5, 23, 29].”
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining (2021, neutral): “Therefore, there is a clear need for a fast, accurate, and reproducible automated method, for segmentation of brain scans of any contrast and resolution, and that can adapt to a wide range of populations.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, methodology): “There are many other methods in the machine learning literature that aim to reduce data labeling efforts, such as transfer learning [179], domain adaptation [180]–[184], and semi-supervised learning [185].”
Measuring Domain Shift for Deep Learning in Histopathology (2020, methodology): “Another technique, termed domain randomization, aim at accomplishing domain generalization by forcing a deeper understanding of images by creating non-realistic variations of image compositions [30], [31].”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, methodology): “Recent work [14] demonstrated the ability of DR to achieve state-of-the-art in 2D bounding box detection of cars in the KITTI dataset [15].”