Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

By F. Saleh, Mohammad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, José Manuel Álvarez. 2018

164 citations recorded by the source corpus.

Training a deep network to perform semantic segmentation requires large amounts of labeled data. To alleviate the manual effort of annotating real images, researchers have investigated the use of synthetic data, which can be labeled automatically. Unfortunately, a network trained on synthetic data performs relatively poorly on real images. While this can be addressed by domain adaptation, existing methods all require having access to real images during training. In this paper, we introduce a drastically different way to handle synthetic images that does not require seeing any real images at training time. Our approach builds on the observation that foreground and background classes are not affected in the same manner by the domain shift, and thus should be treated differently. In particular, the former should be handled in a detection-based manner to better account for the fact that, while their texture in synthetic images is not photo-realistic, their shape looks natural. Our experiments evidence the effectiveness of our approach on Cityscapes and CamVid with models trained on synthetic data only.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

PixMatch: Unsupervised Domain Adaptation via Pixelwise Consistency Training (2021, methodology): “Even for annotations that can be performed by nonexperts like parsing an urban scene into familiar objects, as in the Cityscapes dataset [15], it takes an estimated 90 minutes to annotate a single image [43].”
Bidirectional Learning for Domain Adaptation of Semantic Segmentation (2019, background): “In [31], foreground and background classes are separately treated for decreasing the domain shift respectively.”
ACE: Adapting to Changing Environments for Semantic Segmentation (2019, unknown): “Saleh et al . handle foreground classes using detection methods when addressing domain shift [46].”
DLOW: Domain Flow for Adaptation and Generalization (2018, background): “Recently, there is an increasing interest to apply domain adaptation techniques for semantic segmentation from synthetic data to the real scenario [22, 21, 7, 67, 40, 25, 11, 46, 51, 53, 23, 47, 62, 54, 43, 50, 52, 66, 5].”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “Semantic segmentation in DG Based on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43, 60, 53, 6, 69, 18, 58, 37, 65] have been proposed in semantic segmentation, but only a few DG studies [64, 44] address semantic segmentation, as the majority of the DG methods mainly…”
Semi-Supervised Semantic Segmentation With Cross-Consistency Training (2020, methodology): “Various generative and discriminative domain adaptation methods have been proposed for classification [16, 14, 15, 4] and semantic segmentation [21, 58, 44, 24] tasks.”
Domain Adaptation for Structured Output via Discriminative Patch Representations (2019, background): “, scene layout) across domains; 2) pixel-level adaptation synthesizes target samples [15, 27, 43, 47] to reduce the domain gap during training the segmentation model; 3) pseudo-label re-training [34, 50] generates pseudo ground truth of target images to finetune the model trained on the source domai”
Synthetic Data for Deep Learning (2019, methodology): “[515] presented a Unity3D framework called VEIS (Virtual Environment for Instance Segmentation); while not very realistic, it worked well with their detection-based pipeline (see Section 2.”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, background): “[56] exploit the observation that shape is less affected by the domain gap than appearance for scene semantic segmentation.”