Free Supervision from Video Games

By Philipp Krahenbuhl. 2018

98 citations recorded by the source corpus.

Deep networks are extremely hungry for data. They devour hundreds of thousands of labeled images to learn robust and semantically meaningful feature representations. Current networks are so data hungry that collecting labeled data has become as important as designing the networks themselves. Unfortunately, manual data collection is both expensive and time consuming. We present an alternative, and show how ground truth labels for many vision tasks are easily extracted from video games in real time as we play them. We interface the popular MicrosoftÂ® DirectXÂ® rendering API, and inject specialized rendering code into the game as it is running. This code produces ground truth labels for instance segmentation, semantic labeling, depth estimation, optical flow, intrinsic image decomposition, and instance tracking. Instead of labeling images, a researcher now simply plays video games all day long. Our method is general and works on a wide range of video games. We collected a dataset of 220k training images, and 60k test images across 3 video games, and evaluate state of the art optical flow, depth estimation and intrinsic image decomposition algorithms. Our video game data is visually closer to real world images, than other synthetic dataset.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “No existing synthetic dataset or simulator addresses all of these limitations, including those that have been developed for outdoor scene understanding [7, 18, 21, 32, 36, 39, 53, 54, 55, 56, 61, 72, 73].”
Incremental Object Learning From Contiguous Views (2019, background): “Examples include purposebuilt autonomous driving simulators such as TORCS [47] and CARLA [13], and efforts to leverage commercial video games [25, 40, 39].”
Joint Monocular 3D Vehicle Detection and Tracking (2018, methodology): “Our data collection and annotation pipeline extend the previous works like VIPER [36] and FSV [22], especially in terms of linking identities across frames.”
SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud (2018, methodology): “We use the framework in [31] to generate depth semantic segmentation maps, and use the method in [29] to do Image-LiDAR registration in GTAV.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “Only a few work explored self-supervised learning from synthetic data by using GAN to bridge the domain gap [30], [166].”
Attentive Single-Tasking of Multiple Tasks (2019, methodology): “Validation on additional datasets: We validate our approach in two additional datasets, NYUD [42] and FSV [29].”
Enhancing Photorealism Enhancement (2021, methodology): “game, we follow recent approaches to extracting rendering resources from computer games [27], [87], [93].”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, methodology): “…been used for optical ﬂow [42, 41, 9, 7, 69, 81, 58], robustness estimation of features [51], visual odometry [35, 81], hand tracking [87], shape from shading [82], semantic segmentation [83, 50, 81, 58], amodal segmentation [52, 24], multi-view stereopsis [47] and human pose estimation [89, 26].”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “The community has been investing significant effort in creating high-quality synthetic content, ranging from driving scenes [38, 11, 36, 8, 34, 47, 2], indoor scenes [48, 50, 33], household robotics [35, 22], robotic control [44], game playing [5], optical flow estimation [6, 23], and quadcopter con”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “However, the automatic creation of textual input augmentation remains a signiﬁcant challenge, as a single token can reverse the meaning of a sentence.”
Deep Object-Centric Policies for Autonomous Driving (2018, unknown): “Speciﬁcally, we use the Grand Theft Auto V simulation [19] and the Berkeley DeepDrive Video dataset [5] for online and ofﬂine evaluation, respectively.”
Causal Confusion in Imitation Learning (2019, methodology): “[56] for learning to drive in near-photorealistic GTA-V [24] environments, using behavior cloning with DAggerinspired expert perturbation.”