Augmented Reality meets Deep Learning

By Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars M. Mescheder, Andreas Geiger, Carsten Rother. 2017

24 citations recorded by the source corpus.

The success of deep learning in computer vision is based on the availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Unfortunately, creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment these images with virtual objects. This allows us to create realistic composite images which exhibit both realistic background appearance as well as a large number of complex object arrangements. In contrast to modeling complete 3D environments, our data augmentation approach requires only a few user interactions in combination with 3D shapes of the target object category. We demonstrate the utility of the proposed approach for training a state-of-the-art high-capacity deep model for semantic instance segmentation. In particular, we consider the task of segmenting car instances on the KITTI dataset which we have annotated with pixel-accurate ground truth. Our experiments demonstrate that models trained on augmented imagery generalize better than those trained on synthetic data or models trained on limited amounts of annotated real data.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Driving Policy Transfer via Modularity and Abstraction (2018, methodology): “ld has been studied extensively in computer vision and robotics. Synthetic data has been used for training and evaluation of perception systems in indoor environments [54,18,30] and driving scenarios [40,15, 29,38,45,22,47,2]. Direct transfer from simulation to reality remains difﬁcult even given hi”
On Pre-Trained Image Features and Synthetic Images for Deep Learning (2017, background): “hly elaborate pipelines that are difﬁcult to engineer and need domain speciﬁc expertise [20]. Furthermore, additional effort is needed to collect environment information like illumination information [1] to produce photo-realistic scenes. For real scene composition, one also needs to parse real back”
Geometric Image Synthesis (2018, methodology): “In the first, we utilize an augmented reality dataset where synthetic vehicles were realistically rendered into a scene using “Cycles renderer” from Blender [1].”
All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda (2021, background): “ing [220], [221], computer vision [222], [223], and recommender systems [224], [225].”
A survey on face data augmentation for the training of deep neural networks (2019, background): “More experiences of data augmentation by AR can be found in [133], [134], and [135].”
Semantic Foggy Scene Understanding with Synthetic Data (2017, background): “on and object detection for real foggy urban scenes, by adding synthetic fog to real images taken under clear weather. Hence, our approach is based on only partially synthetic data. In the same vein, [2] is based on real urban scenes, augmented with virtual cars. A very interesting project is \FOG&q”