VirtualWorlds as Proxy for Multi-object Tracking Analysis

By Adrien Gaidon, Qiao Wang, Yohann Cabon, E. Vig. 2016

1,225 citations recorded by the source corpus.

Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called "Virtual KITTI", automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “[5] show that pre-training a deep neural network on synthetic data leads to improved performance.”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “No existing synthetic dataset or simulator addresses all of these limitations, including those that target outdoor scene understanding [8, 20, 23, 34, 38, 41, 55, 56, 58, 60, 65, 76, 77].”
Learning To Simulate (2018, background): “It is, thus, natural for recent works to explore simulation as a cheaper alternative to human annotation (Gaidon et al., 2016; Ros et al., 2016; Richter et al., 2016).”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “…et al., 2016, Bernuth et al., 2018] or collecting real [Wen et al., 2015, Yu et al., 2018, Che et al., 2019, Caesar et al., 2019] and artiﬁcial [Gaidon et al., 2016, Ros et al., 2016, Richter et al., 2017, Johnson-Roberson et al., 2017] datasets with varying weather conditions, no general…”
Driving Policy Transfer via Modularity and Abstraction (2018, methodology): “ld has been studied extensively in computer vision and robotics. Synthetic data has been used for training and evaluation of perception systems in indoor environments [54,18,30] and driving scenarios [40,15, 29,38,45,22,47,2]. Direct transfer from simulation to reality remains difﬁcult even given hi”
CARLA: An Open Urban Driving Simulator (2017, background): “Custom simulation setups are commonly used to train and benchmark robotic vision systems [2, 9, 10, 11, 20, 25, 27, 29].”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “SYNTHIA [16], Virtual KITTI [9], TORCS [21], and CARLA [7] provide synthetic data for autonomous driving.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, methodology): “Screenshots of our framework and other simulated learning frameworks: ALE [36], ViZDoom [37], UETorch [38], Project Malmo [39], SceneNet [40], TORCS [41], SYNTHIA [42], Virtual KITTI [43].”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, methodology): “providing raw data without semantic labels. An alternative to collecting real-world multimodal driving data is by generating synthetic data via simulation. CARLA [16], SYNTHIA [47], and Virtual KITTI [21] simulate virtual cities using game engines. Playing for Benchmarks [45] retrieves renderings an”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “have also shown to benefit by using such approaches [11, 17, 40, 41].”
VirtualHome: Simulating Household Activities Via Programs (2018, methodology): “Simulations using game engines have recently been developed to facilitate training visual models for autonomous driving [7, 19, 6], quadcopter flying [21], or other robotic tasks [5].”
Playing for Benchmarks (2017, methodology): “We sampled 500 random images each from VIPER, SYNTHIA [55], Virtual KITTI [17], the Frei-burg driving sequence [41], and the Urban Canyon dataset (regular perspective camera) [64], as well as Cityscapes as a real-world reference.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “on recognition in videos. (Image courtesy C. De Souza). virtual worlds with diverse, realistic, and physically plausible events and actions. Popular virtual words are SYNTHIA37 [176], Virtual KITTI38 [177] and GTA-V [178] (see also Figure 23). Such virtually generated and controlled environments com”
Neural Scene De-rendering (2017, background): “In contrast, graphics engines have been rather mature, especially for virtual environments [10, 38].”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “Virtual environments have been a remedy for this, classically [64] and today [100, 34, 29, 79, 44, 10, 8, 68, 92].”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, background): “As a large amount of annotated data is crucial in order to train such deep networks, multiple new datasets have been released to encourage further research in this area, including Synthia [45], Virtual KITTI [18], and Cityscapes [11].”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “There also exist several synthetic urban datasets [17], [32], [82], [86].”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “The computer vision community has used synthetic data for many tasks, including object recognition [23, 44, 51, 73], scene understanding [12, 25, 47, 50], eye tracking [63, 68], hand tracking [40, 61], and full-body analysis [41, 59, 65].”
Sparsity Invariant CNNs (2017, background): “While evaluation on synthetic datasets [45, 11, 39] is possible, it remains an open question if the level of realism attained by such datasets is sufficient to conclude about an algorithm’s performance in challenging real-world situations.”
An Evaluation Metric for Object Detection Algorithms in Autonomous Navigation Systems and its Application to a Real-Time Alerting System (2018, methodology): “When we use the network and the scoring measure on an image from Virtual KITTI dataset [6] we clearly observe in Fig.”