Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes

By Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars M. Mescheder, Andreas Geiger, Carsten Rother. 2017

485 citations recorded by the source corpus.

The success of deep learning in computer vision is based on the availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Unfortunately, creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation and object detection models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment these images with virtual objects. In contrast to modeling complete 3D environments, our data augmentation approach requires only a few user interactions in combination with 3D models of the target object category. Leveraging our approach, we introduce a novel dataset of augmented urban driving scenes with 360 degree images that are used as environment maps to create realistic lighting and reflections on rendered objects. We analyze the significance of realistic object placement by comparing manual placement by humans to automatic methods based on semantic scene analysis. This allows us to create composite images which exhibit both realistic background appearance as well as a large number of complex object arrangements. Through an extensive set of experiments, we conclude the right set of parameters to produce augmented data which can maximally enhance the performance of instance segmentation models. Further, we demonstrate the utility of the proposed approach on training standard deep models for semantic instance segmentation and object detection of cars in outdoor driving scenarios. We test the models trained on our augmented data on the KITTI 2015 dataset, which we have annotated with pixel-accurate ground truth, and on the Cityscapes dataset. Our experiments demonstrate that the models trained on augmented imagery generalize better than those trained on fully synthetic data or models trained on limited amounts of annotated real data.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, background): “[12] overlay synthetic objects on real backgrounds and Lee et al.”
Synthetic Data for Deep Learning (2019, background): “In [5], the backgrounds come from the KITTI dataset of outdoor scenes and the objects are synthetic models of cars, while in [198] the authors place synthetic objects into indoor scenes with an eye towards home service robots.”
Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms (2021, background): “12 and Table V present the qualitative and quantitative results on the KITTI semantic segmentation dataset [40], respectively.”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “The ideas of those methods include leveraging a graphics engine [1, 243], exploiting texture-mapped surfels [341], leveraging real-world data [48], and learning a controllable neural simulator [115].”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, methodology): “The number of training epochs is set to 180 for Cityscapes, 120 for Camvid and 90 for KITTI.”
Geometric Image Synthesis (2018, methodology): “[1,2] consider the problem of augmenting scenes from the KITTI dataset with virtual vehicles for improving object detectors and instance segmentation algorithms.”
Deep Learning for Intelligent Transportation Systems: A Survey of Emerging Trends (2020, methodology): “Building on image classification as the source task, many applications such as traffic sign recognition [78] or semantic segmentation [6] are prime for transfer learning.”
ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape (2018, background): “Since annotating monocular data with metrically accurate 3D annotations is usually costly and difﬁcult, many recent works leverage synthetic data [10, 8, 2, 1] to train their methods [20, 17, 35].”
FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, unknown): “It plays a fundamental role in scene understanding and is of great importance to real-world applications, such as autonomous driving [7], medical imaging [44], augmented reality [1], etc.”