Playing for Data: Ground Truth from Computer Games

By Stephan R. Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun. 2016

2,267 citations recorded by the source corpus.

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just $\tfrac{1}{3}$ of the CamVid training set outperform models trained on the complete CamVid training set.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Combining data from high-quality simulators with other approaches like ﬁne-tuning can also reduce the number of labeled samples required in the real world [34].”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “, 2020), as well as from synthetic data to real data (including sim2real) (Wang et al., 2015; Ganin & Lempitsky, 2015; Richter et al., 2016; Peng et al., 2018; Hellendoorn et al., 2019; Xu et al., 2020; Bellemare et al., 2020), as synthetic data can be easier to obtain than real data.”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, background): “More recently, it has been shown that deep networks can also boost their performance when pretrained on data other than natural images, such as video games [26, 28] or clipart [3], and this an interesting direction to explore.”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “In addition to providing a mesh painting interface that operates at the granularity of object parts [8, 19, 53, 55, 72], our tool has two unique features that are especially helpful for annotating our scenes.”
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks (2018, background): “However, both [4] and [17] demonstrate that complete realism is not necessary to improve results with synthetic data.”
Learning To Simulate (2018, methodology): “Many alternative simulators like (Mueller et al., 2017; Richter et al., 2016; Shah et al., 2018) are similar where an agent can navigate a few pre-deﬁned maps, but the scene itself is not parameterized and can be changed on the ﬂy.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Semantic segmentation is a fundamental problem in computer vision where significant progress has been made thanks to the surge of deep learning [13–15] and the availability of large-scale human-annotated or synthetic datasets [4, 17, 23, 40, 55].”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “Datasets and Evaluation Metrics: We validate MAS(3) on the standard GTA5 (Richter et al. 2016)→Cityscapes (Cordts et al.”
Driving Policy Transfer via Modularity and Abstraction (2018, methodology): “ld has been studied extensively in computer vision and robotics. Synthetic data has been used for training and evaluation of perception systems in indoor environments [54,18,30] and driving scenarios [40,15, 29,38,45,22,47,2]. Direct transfer from simulation to reality remains difﬁcult even given hi”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, positive): “Our findings in this regard are aligned with the results obtained in other recent works [32], which also used only synthetic renderings to train visual models, but did not explicitly consider wide-ranging randomization of the training scenes.”
CARLA: An Open Urban Driving Simulator (2017, methodology): “And commercial games have been used to acquire high-ﬁdelity data for training and benchmarking visual perception systems [23, 24].”
Semantic Scene Completion from a Single Depth Image (2016, background): “Although recent works have been focusing on generating segmentation labels for 2D image through rendering synthetic scenes [8, 27], the 3D aspect of such data has not been fully utilized.”
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (2016, background): “Consequently, there has been a renewed interest in training models in the synthetic domain and applying them in real–world settings [8, 47, 38, 42, 25, 32, 35, 37].”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, methodology): “We visualize the results of image-space adaptation between GTA5 and Cityscapes in Figure 6.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Visual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47, 48, 70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban”