Play and Learn: Using Video Games to Train Computer Vision Models

By Alireza Shafaei, James J. Little, Mark W. Schmidt. 2016

116 citations recorded by the source corpus.

Video games are a compelling source of annotated data as they can readily provide fine-grained groundtruth for diverse tasks. However, it is not clear whether the synthetically generated data has enough resemblance to the real-world images to improve the performance of computer vision models in practice. We present experiments assessing the effectiveness on real-world data of systems trained on synthetic RGB images that are extracted from a video game. We collected over 60000 synthetic samples from a modern video game with similar conditions to the real-world CamVid and Cityscapes datasets. We provide several experiments to demonstrate that the synthetically generated RGB images can be used to improve the performance of deep neural networks on both image segmentation and depth estimation. These results show that a convolutional network trained on synthetic data achieves a similar test error to a network that is trained on real-world data for dense image classification. Furthermore, the synthetically generated RGB images can provide similar or better results compared to the real-world datasets if a simple domain adaptation technique is applied. Our results suggest that collaboration with game developers for an accessible interface to gather data is potentially a fruitful direction for future work in computer vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Human pose estimation with Kinect [35] and, more recently, a plethora of other tasks have been tackled using synthetic data [43, 42, 29, 34].”
Playing for Data: Ground Truth from Computer Games (2016, background): “Very recent concurrent work [41, 51] demonstrates the heightened interest in using virtual worlds to train algorithms for scene understanding.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Visual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47, 48, 70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “…have great promise for deep learning across a breadth of computer vision problems, including optical ﬂow [148, 24, 156, 147], object trackers [225, 91], depth estimation from RGB [192], object detection [144, 234, 253] semantic segmentation [109, 179, 176] or human actions recognition [211].”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, methodology): “This has encouraged the development of several synthetic scene projects with associated datasets, such as Carla [8], SYNTHIA [31], and others [35, 30].”
TorontoCity: Seeing the World with a Million Eyes (2016, methodology): “A popular alternative is to use synthetic data to generate large scale benchmarks [4, 6, 27, 30, 32, 14, 7, 29].”
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes (2017, background): “Upon observing the obvious mismatch between virtual and real data [51, 47, 48], we expect domain adaptation to enhance the segmentation performance on real images by networks trained on virtual ones.”
Domain Adaptation for Semantic Segmentation of Urban Scenes (2017, background): “Upon observing the obvious mismatch between virtual and real data [51, 47, 48], we expect domain adaptation to enhance the segmentation performance on real images by networks trained on virtual ones.”
Augmented Reality meets Deep Learning (2017, background): “A common strategy to circumvent the need for human labels is to train neural networks on synthetic data obtained from a 3D renderer for which ground truth labels can be automatically obtained [8, 11, 16, 20, 21, 23, 29, 31].”
Soccer on Your Tabletop (2018, background): “An alternative would be to use synthetic data [49, 22, 47, 44], but these virtual worlds are far from our soccer scenario.”
Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training (2017, background): “The obstacles associated with developing a large dataset of real images can be circumvented by generating synthetic images [22, 26, 24].”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, background): “[30] who are using an undisclosed game engine, and Qui et al.”
Enhancing Photorealism Enhancement (2021, background): “Although G-buffers generally contain no explicit semantic information, they are consistent with semantic entities [27], [91].”
Cross-Domain Self-Supervised Multi-task Feature Learning Using Synthetic Imagery (2017, background): “Synthetic data has also begun to show promising usage for vision tasks including learning optical flow [43], semantic segmentation [56, 57, 59], video analysis [20], stereo [75], navigation [80], and intuitive physics [39, 70, 47].”
SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? (2017, background): “[28] produced synthetic training data from photorealistic gaming engines, validating the performance on real-world segmentation tasks.”
Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes (2017, background): “Work that rendered synthetic 3D images has focused on domains where the camera viewpoint is highly constrained (e.g., driving [30, 33, 29]), has used manually speciﬁed camera trajectories (e.g., by recording ﬁrst-person user trajectories in the virtual scene, or from camera paths in the real world…”
Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes (2017, background): “A common strategy to circumvent the need for human labels is to train neural networks on synthetic data obtained from a 3D renderer for which ground truth labels can be automatically obtained, ([23, 20, 15, 29, 31, 21, 11, 8]).”