Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?

By Matthew Johnson-Roberson, Charlie Barto, Rounak Mehta, S. N. Sridhar, Karl Rosaen, Ram Vasudevan. 2016

714 citations recorded by the source corpus.

Deep learning has rapidly transformed the state of the art algorithms used to address a variety of problems in computer vision and robotics. These breakthroughs have relied upon massive amounts of human annotated training data. This time consuming process has begun impeding the progress of these deep learning efforts. This paper describes a method to incorporate photo-realistic computer images from a simulation engine to rapidly generate annotated data that can be used for the training of machine learning algorithms. We demonstrate that a state of the art architecture, which is trained only using these synthetic annotations, performs better than the identical architecture trained on human annotated real-world data, when tested on the KITTI data set for vehicle detection. By training machine learning algorithms on a rich virtual world, real objects in real scenes can be learned and classified using synthetic data. This approach offers the possibility of accelerating deep learning's application to sensor-based classification problems like those that appear in self-driving cars. The source code and data to train and validate the networks described in this paper are made available for researchers.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “…or collecting real [Wen et al., 2015, Yu et al., 2018, Che et al., 2019, Caesar et al., 2019] and artiﬁcial [Gaidon et al., 2016, Ros et al., 2016, Richter et al., 2017, Johnson-Roberson et al., 2017] datasets with varying weather conditions, no general solution towards the problem has yet emerged.”
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (2016, background): “Unfortunately, models naively trained on synthetic data do not typically generalize to real images.”
Domain Adaptive Faster R-CNN for Object Detection in the Wild (2018, methodology): “We conduct extensive experiments to evaluate our Domain Adaptive Faster R-CNN using multiple datasets including Cityscapes [5], KITTI [17], SIM 10k [30], etc.”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “Recent advances in computer graphics have stimulated the interests to train deep learning models on photo-realistic synthetic data with computergenerated annotations, and apply them in real-world settings [12, 29, 47, 37, 41, 48, 20, 19, 32, 22, 34, 33].”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “(62) offered a discussion of whether virtual worlds may replace human-generated annotations for real-world tasks.”
Adversarial Geometry and Lighting using a Differentiable Renderer (2018, background): “[16] Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan.”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “[13] used the Grand Theft Auto (GTA) engine from Rockstar Games, and Ros et al.”
Harmonizing Transferability and Discriminability for Adapting Object Detectors (2020, methodology): “Sim10K [22] is a dataset produced based on the computer game Grand Theft Auto V (GTA V).”
Strong-Weak Distribution Alignment for Adaptive Object Detection (2018, methodology): “We evaluate our approach on four domain shifts– PASCAL [8] to Clipart [15], PASCAL to Watercolor [15], Cityscapes [6] to FoggyCityscapes [34], and GTA [16] to Cityscapes–to demonstrate that it is effective for adaptation between both dissimilar and similar domains.”
SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud (2017, methodology): “In the autonomous driving community, the video game Grand Theft Auto has been used to retrieve data for object detection and segmentation [19], [20].”
Learning Pedestrian Detection from Virtual Worlds (2019, methodology): “The GTA graphic engine is not publicly available, so it is not easy to extract the detailed masks around each pedestrian instance; [23] overcame this issue by extracting semantic masks and separating the instances by exploiting depth information.”
Learning From Synthetic Data for Crowd Counting in the Wild (2019, methodology): “Previous synthetic GTA5 datasets [28, 16, 27] capture normal scenes directed by the game programming.”
Soccer on Your Tabletop (2018, background): “An alternative would be to use synthetic data [49, 22, 47, 44], but these virtual worlds are far from our soccer scenario.”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, methodology): “[16] used synthetic data captured from GTA V to train an object detector for cars and demonstrated that photo-realism aided in the training of DNNs.”
LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving (2020, methodology): “Researchers have used games such as Grand Theft Auto V to generate synthetic datasets [28], [29], [30].”
Geometric Image Synthesis (2018, background): “[23] have been the first to demonstrate that content from commercial video games can be accessed for collecting semantic segmentation, optical flow and object detection ground truth by “playing”.”