Playing for Benchmarks

By Stephan R. Richter, Zeeshan Hayder, Vladlen Koltun. 2017

518 citations recorded by the source corpus.

We present a benchmark suite for visual perception. The benchmark is based on more than 250K high-resolution video frames, all annotated with ground-truth data for both low-level and high-level vision tasks, including optical flow, semantic instance segmentation, object detection and tracking, object-level 3D scene layout, and visual odometry. Ground-truth data for all tasks is available for every frame. The data was collected while driving, riding, and walking a total of 184 kilometers in diverse ambient conditions in a realistic virtual world. To create the benchmark, we have developed a new approach to collecting ground-truth data from simulated worlds without access to their source code or content. We conduct statistical analyses that show that the composition of the scenes in the benchmark closely matches the composition of corresponding physical environments. The realism of the collected data is further validated via perceptual experiments. We analyze the performance of state-of-the-art methods for multiple tasks, providing reference baselines and highlighting challenges for future research.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “No existing synthetic dataset or simulator addresses all of these limitations, including those that target outdoor scene understanding [8, 20, 23, 34, 38, 41, 55, 56, 58, 60, 65, 76, 77].”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “…or collecting real [Wen et al., 2015, Yu et al., 2018, Che et al., 2019, Caesar et al., 2019] and artiﬁcial [Gaidon et al., 2016, Ros et al., 2016, Richter et al., 2017, Johnson-Roberson et al., 2017] datasets with varying weather conditions, no general solution towards the problem has yet…”
A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, methodology): “Some researchers even used computer games such as Grand Theft Auto V [294].”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39, 40, 41, 42, 43], autonomous driving [44, 45, 46], grasping [47, 48, 49], single-task manipulation [50], among others.”
CARLA: An Open Urban Driving Simulator (2017, methodology): “And commercial games have been used to acquire high-ﬁdelity data for training and benchmarking visual perception systems [23, 24].”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “Multi-task learning has experienced recent progress and the reported advantages are another support for existence of a useful structure among tasks [90, 97, 48, 73, 70, 48, 16, 94, 58, 9, 63].”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, methodology): “ real-world multimodal driving data is by generating synthetic data via simulation. CARLA [16], SYNTHIA [47], and Virtual KITTI [21] simulate virtual cities using game engines. Playing for Benchmarks [45] retrieves renderings and annotations from GTA without access to their source code. These have t”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, methodology): “To ensure annotation quality, we embed ‘sentinel’ tasks within each batch of work [6, 12, 29], constituting at least 10% of each batch.”
Unlimited Road-scene Synthetic Annotation (URSA) Dataset (2018, methodology): “as object detection [14], [15], and semantic segmentation [11], [15].”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “By harnessing photo-realistic simulation of urban environments such as Grand Thief Auto (GTA), practically an unlimited amount of synthetic scene images can be generated for training deep learning models [35, 34, 33].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “VIPER-VPS and Cityscapes-VPS are aware of VPS evaluation, but VIPER-VPS is a synthesized dataset.”
Soccer on Your Tabletop (2018, background): “It turns out that while playing Electronic Arts FIFA games and intercepting the calls between the game engine and the GPU [43, 42], it is possible to extract depth maps from video game frames.”
Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, methodology): “The source is GTA (Richter et al., 2017), a video game in an urban environment, and the target is Cityscapes (Cordts et al., 2016), an urban autonomous driving dataset.”
Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach (2018, methodology): “rming alternative to supervise semantic segmentation models. Recent advances in computer graphics make it possible to generate synthetic images, and per-pixel annotations from virtual 3D environments [39,38]. Training on synthetic data seems to be a tempting way to reduce manual labelling cost, howe”
Joint Monocular 3D Vehicle Detection and Tracking (2018, methodology): “Virtual KITTI [11] D,T C 21k 2610 181k X X X X VIPER [36] D,T C,P 184k 8272 5203k X X X X Ours D,T C,P 688k 325k 10068k X X X X”
Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception (2019, background): “Playing for Benchmarks [14] generates 250,000 video frames of seman-”
Learning to Drive from Simulation without Real World Labels (2018, background): “While recent simulated environments, such as [24], [5], have benefited from continued progress in photo realistic gaming engines, the reality gap remains challenging.”