Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer

By Jun Xie, Martin Kiefel, Ming-Ting Sun, Andreas Geiger. 2015

172 citations recorded by the source corpus.

Semantic annotations are vital for training models for object recognition, semantic segmentation or scene understanding. Unfortunately, pixelwise annotation of images at very large scale is labor-intensive and only little labeled data is available, particularly at instance level and for street scenes. In this paper, we propose to tackle this problem by lifting the semantic instance labeling task from 2D into 3D. Given reconstructions from stereo or laser data, we annotate static 3D scene elements with rough bounding primitives and develop a model which transfers this information into the image domain. We leverage our method to obtain 2D labels for a novel suburban video dataset which we have collected, resulting in 400k semantic and instance image annotations. A comparison of our method to state-of the-art label transfer baselines reveals that 3D information enables more efficient annotation while at the same time resulting in improved accuracy and time-coherent labels.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, background): “Very recently, [75] announced a new semantic scene labeling dataset for suburban traffic scenes.”
Playing for Data: Ground Truth from Computer Games (2016, methodology): “Classes were defined to be compatible with other semantic segmentation datasets for outdoor scenes [8, 11, 39, 52].”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, methodology): “For example, researchers choose to utilize 3D information [37], rendered images [30, 31], or weakly supervised labels [32, 34, 3] for labeling.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “In contrast, Xie et al. (2016) reason jointly about all objects in the scene and also handle categories for which CAD models or 3D point measurements are unavailable.”
Augmented Reality meets Deep Learning (2017, methodology): “KITTI-360 For our experiments, we created a new dataset which contains 200 images from the dataset presented in [30].”
The ApolloScape Dataset for Autonomous Driving (2018, methodology): “The basic idea of our pipeline is similar to the one described in [13], while some key techniques used in our pipeline are different.”
LiDAR-Video Driving Dataset: Learning Driving Policies Effectively (2018, background): “In consequence, it is feasible to segment point clouds in unsupervised ways [2, 26].”
Learning 3D Shape Completion from Laser Scan Data with Weak Supervision (2018, unknown): “, [31,50]).”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, background): “Since we are labeling each point, we are able to annotate objects, even with complex occlusions, more precisely than just using bounding volumes [11].”
Find your own way: Weakly-supervised segmentation of path proposals for urban autonomy (2016, methodology): “A recent method uses sparse 3D prior information to transfer labels to real-world 2D images [24] but requires sophisticated 3D reconstructions and manual 3D annotations.”
The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes (2017, methodology): “Another recent trend involves integrating synthetically-rendered data from sources like the Grand Theft Auto V engine [45], Synthia [46] or semantic instance labeling via 3D to 2D label transfer [56].”