ROAD: Reality Oriented Adaptation for Semantic Segmentation of Urban Scenes

By Yuhua Chen, Wen Li, Luc Van Gool. 2017

316 citations recorded by the source corpus.

Exploiting synthetic data to learn deep models has attracted increasing attention in recent years. However, the intrinsic domain difference between synthetic and real images usually causes a significant performance drop when applying the learned model to real world scenarios. This is mainly due to two reasons: 1) the model overfits to synthetic images, making the convolutional filters incompetent to extract informative representation for real images; 2) there is a distribution difference between synthetic and real data, which is also known as the domain adaptation problem. To this end, we propose a new reality oriented adaptation approach for urban scene semantic segmentation by learning from synthetic data. First, we propose a target guided distillation approach to learn the real image style, which is achieved by training the segmentation model to imitate a pretrained real style model using real images. Second, we further take advantage of the intrinsic spatial structure presented in urban scene images, and propose a spatial-aware adaptation scheme to effectively align the distribution of two domains. These two modules can be readily integrated with existing state-of-the-art semantic segmentation networks to improve their generalizability when adapting from synthetic to real urban scenes. We evaluate the proposed method on Cityscapes dataset by adapting from GTAV and SYNTHIA datasets, where the results demonstrate the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recognition in Terra Incognita (2018, neutral): “[45,46,47] investigate methods of domain adaptation for semantic segmentation, focusing mainly on cars and pedestrians and either adapting from synthetic to real data, from urban to suburban scenes, or from PASCAL to a camera on-board a car.”
Domain Adaptive Faster R-CNN for Object Detection in the Wild (2018, background): “Recently there are some works concerning tasks such as semantic segmentation [4, 27, 61], fine-grained recognition [16] etc.”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, methodology): “In [4], adversarial training is used for spatial-aware adaptation along with a distillation loss to speciﬁcally address synthetic-2-real domain shift.”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “Semantic segmentation in DG Based on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43, 60, 53, 6, 69, 18, 58, 37, 65] have been proposed in semantic segmentation, but only a few DG studies [64, 44] address semantic segmentation, as the majority of the DG methods mainly…”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “[40] develop a distillation loss in addition to performing location-aware alignment (e.”
Knowledge Transfer Graph for Deep Collaborative Learning (2019, methodology): “Distillation has been applied to object detection [4], domain adaptation [5], text-to-speech [21], etc.”
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation (2019, methodology): “We compare our proposed method with Curriculum DA [51], CyCADA [14], MCD [39], LSD-seg [41], AdaptSegNet [44], ROAD [5], Conservative Loss [55], DCAN [47], and CBST [57].”
Spatial Attention Pyramid Network for Unsupervised Domain Adaptation (2020, unknown): “6 ROAD [3] 76.”
Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection (2019, background): “As unsupervised domain adaptation attracted considerable interest with its effectiveness, recent works [17, 47, 6, 5, 38, 43, 19, 48] have been attempted to address the generalization issue in the semantic segmentation task.”
Adapting Object Detectors via Selective Cross-Domain Alignment (2019, background): “Other efforts [5, 9, 20, 49, 41], instead focus on the learning of domain-invariant features.”
Exploring Object Relation in Mean Teacher for Cross-Domain Detection (2019, background): “, object detection [4, 35] and semantic segmentation [5, 16, 53].”
Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization (2020, unknown): “timization procedure. The general idea of domain adaption is to align the source domain and target domain distributions in the input level [3,19], semantic feature level [9,34], or output space level [5,39,40,43,45]. Most methods adopt Generative Adversarial Networks and achieve better performance o”
FDA: Fourier Domain Adaptation for Semantic Segmentation (2020, unknown): “1 ROAD[7] 77.”
Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation (2022, methodology): “Most existing CDSS methods [11, 18, 29, 34, 42, 44] seek to learn domain-invariant representations via adversarial training to align marginal distributions (p(x)) or conditional distributions (p(x|y)).”
PixMatch: Unsupervised Domain Adaptation via Pixelwise Consistency Training (2021, background): “Most of the literature is concerned with adversarial training [23, 49, 52, 11].”
Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes (2019, methodology): “To measure the performance, we compared the predictions on the Cityscapes validation set with the ground truth labels and computed the mean Intersection over Union (mIoU) as done by most competing approaches [12, 4, 35].”
DLOW: Domain Flow for Adaptation and Generalization (2018, methodology): “Recently, there is an increasing interest to apply domain adaptation techniques for semantic segmentation from synthetic data to the real scenario [22, 21, 7, 67, 40, 25, 11, 46, 51, 53, 23, 47, 62, 54, 43, 50, 52, 66, 5].”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “The other four work [21, 45, 9, 8] focus on domain adaptation to enhance the segmentation performance on real images by networks trained on virtual ones, which is also our concern in this work.”