Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

By Yuhua Chen, Wen Li, Xiaoran Chen, Luc Van Gool. 2018

276 citations recorded by the source corpus.

As an alternative to manual pixel-wise annotation, synthetic data has been increasingly used for training semantic segmentation models. Such synthetic images and semantic labels can be easily generated from virtual 3D environments. In this work, we propose an approach to cross-domain semantic segmentation with the auxiliary geometric information, which can also be easily obtained from virtual environments. The geometric information is utilized on two levels for reducing domain shift: on the input level, we augment the standard image translation network with the geometric information to translate synthetic images into realistic style; on the output level, we build a task network which simultaneously performs semantic segmentation and depth estimation. Meanwhile, adversarial training is applied on the joint output space to preserve the correlation between semantics and depth. The proposed approach is validated on two pairs of synthetic to real dataset: from Virtual KITTI to KITTI, and from SYNTHIA to Cityscapes, where we achieve a clear performance gain compared to the baselines and various competing methods, demonstrating the effectiveness of the geometric information for cross-domain semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Domain Adaptation for Traffic Density Estimation (2021, background): “The authors of (Chen et al., 2019) combine semantic segmentation and depth estimation to boost the adaptation performance, providing to the discriminator the segmentation and the depth prediction maps jointly.”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “We have compared our performance against the adversarial learning-based UDA methods: GIO-Ada (Chen et al. 2019b), ADVENT (Vu et al. 2019), AdaSegNet (Tsai et al. 2018), TGCF-DA+SE (Choi, Kim, and Kim 2019), PCEDA (Yang et al. 2020), and CyCADA (Hoffman et al. 2018a).”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, unknown): “Several works also combine self-training and adversarial training [35,39,69,93], minimize the entropy [7,67,68], refine boundaries [41], use a curriculum [13,14,89], or learn auxiliary tasks [8,30,68,71].”
Domain Adaptation for Image Dehazing (2020, methodology): “, image classification [10], semantic segmentation [5], and depth prediction [37].”
Confidence Regularized Self-Training (2019, methodology): “Examples of these methods include maximum mean discrepancy (MMD) [33, 62], deep correlation alignment (CORAL) [56], sliced Wasserstein discrepancy [28], adversarial learning at input-level [16, 24, 68], feature level [8, 14, 23, 31, 50, 61, 64], output space level [60], and a variety of follow up wo”
Unbiased Mean Teacher for Cross-domain Object Detection (2020, unknown): “When those trained models are applied to new environments, significant performance drops have often been observed in many computer vision tasks [5, 6, 11, 37, 54].”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”
Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes (2019, background): “…many works addressed the source to target domain shift problem with various techniques, such as cycle consistency [11], GANs [29], output space alignment [35], distillation loss [4], class-balanced self-training [46], conservative loss [45], geometrical guidance [3] and adaptation networks [42].”
Phase Consistent Ecological Domain Adaptation (2020, background): “Further, [6] proposes a geometrically guided adaptation aided with depth in a multitask learning framework.”
Machine Learning for Synthetic Data Generation: a Review (2023, background): “Synthetic data has emerged as a transformative solution in this context, significantly improving the labeling process [54].”
BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation (2021, methodology): “A variety of unsupervised domain adaptation methods are proposed to maximally eliminate the domain discrepancy through adversarial feature learning [22, 6, 13], entropy minimization [52, 53, 5], selftraining [65, 66, 31, 11], etc.”
DeceptionNet: Network-Driven Domain Randomization (2019, unknown): “[7] 78.”
FDA: Fourier Domain Adaptation for Semantic Segmentation (2020, background): “[6] uses dense depth, readily available in synthetic data.”
DLOW: Domain Flow for Adaptation and Generalization (2018, background): “Recently, there is an increasing interest to apply domain adaptation techniques for semantic segmentation from synthetic data to the real scenario [22, 21, 7, 67, 40, 25, 11, 46, 51, 53, 23, 47, 62, 54, 43, 50, 52, 66, 5].”