Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes

By Yang Zhang, Philip David, Boqing Gong. 2017

360 citations recorded by the source corpus.

During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is a core task of various emerging industrial applications such as autonomous driving and medical imaging. However, to train CNNs requires a huge amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNN models on photo-realistic synthetic data with computer-generated annotations. Despite this, the domain mismatch between the real images and the synthetic data significantly decreases the models’ performance. Hence we propose a curriculum-style learning approach to minimize the domain gap in semantic segmentation. The curriculum domain adaptation solves easy tasks first in order to infer some necessary properties about the target domain; in particular, the first task is to learn global label distributions over images and local distributions over landmark superpixels. These are easy to estimate because images of urban traffic scenes have strong idiosyncrasies (e.g., the size and spatial relations of buildings, streets, cars, etc.). We then train the segmentation network in such a way that the network predictions in the target domain follow those inferred properties. In experiments, our method significantly outperforms the baselines as well as the only known existing approach to the same problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recognition in Terra Incognita (2018, neutral): “[45,46,47] investigate methods of domain adaptation for semantic segmentation, focusing mainly on cars and pedestrians and either adapting from synthetic to real data, from urban to suburban scenes, or from PASCAL to a camera on-board a car.”
Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, background): “Step 6 [50,51,52,53,54,55,56,57,58,59] Flowers, Lamp Shade, Spoon, Candles, Clipboards Scissors, TV, Curtains, Fork, Soda”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, background): “Zhang et al. (2017) instead perform segmentation adaptation by aligning label distributions both globally and across superpixels in an image.”
Learning to Adapt Structured Output Space for Semantic Segmentation (2018, methodology): “We present adaptation results in Table 1 with comparisons to the state-of-the-art domain adaptation methods [12, 13, 39].”
Domain Adaptive Faster R-CNN for Object Detection in the Wild (2018, background): “Recently there are some works concerning tasks such as semantic segmentation [4, 27, 61], fine-grained recognition [16] etc.”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, background): “…techniques through multiple losses, [46] combines the generative approach for appearance adaptation and adversarial training for representation adaptation, and [45] propose a curriculum-style learning for UDA by enforcing the consistency on local (superpixel-level) and global label distributions.”
Survey on semantic segmentation using deep learning techniques (2019, background): “Yang et al. [221] proposed a curriculum-style learning approach to minimize the domain gap.”
Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach (2018, methodology): “Following this line, many works have been proposed to address the domain shift problem in semantic segmentation using different techniques, such as curriculum style learning [57], distillation loss [5], output space alignment [53], class-balanced self-training [62], conservative loss [61], etc.”
Phase Consistent Ecological Domain Adaptation (2020, background): “[45] proposes curriculum learning to gradually minimize the domain gap using anchor points.”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, background): “Domain Adaptation Another line of research tackles the shift of unlabeled data distributions by domain adaptation [61, 29, 51, 28, 70, 63].”
Where and How to Transfer: Knowledge Aggregation-Induced Transferability Perception for Unsupervised Domain Adaptation (2021, methodology): “Furthermore, curriculum learning is employed by [27], [48] to predict the category-wise distribution properties of target domain, which gradually bridges the domain-wise distribution discrepancy from the easy-to-adapt samples to the difﬁ-cult ones.”
HGNet: Hybrid Generative Network for Zero-Shot Domain Adaptation (2020, background): “And recent works attempt to apply DA strategy in computer vision like image classiﬁcation [12, 18], object segmentation [29, 36, 26] and image caption [3].”
Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation (2020, neutral): “…(DA) methods are developed to fully utilize previously labeled datasets and unlabeled data on hand in a transductive manner, which have obtained promising results in object recognition (Long et al., 2015; Tzeng et al., 2017), semantic segmentation (Zhang et al., 2017; Hoffman et al., 2018), etc.”
Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation (2019, methodology): “The general concept of curriculum learning has been applied to domain adaptation by ordering tasks [42] or target-domain pixels [47], while we order domains.”
Taking a Closer Look at Domain Shift: Category-Level Adversaries for Semantics Consistent Domain Adaptation (2018, methodology): “Besides, CLAN also outperforms the state-of-the-art methods, which improves the mIOU by over +7% compared with MCD [34], CDA [44] and CyCADA [14].”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “Semantic segmentation in DG Based on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43, 60, 53, 6, 69, 18, 58, 37, 65] have been proposed in semantic segmentation, but only a few DG studies [64, 44] address semantic segmentation, as the majority of the DG methods mainly…”
Adapting Object Detectors via Selective Cross-Domain Alignment (2019, methodology): “A variety of methods have been developed along this line, which have shown encouraging results on image classification [1, 12, 13, 16, 28, 43] and semantic segmentation [20, 39, 42, 46, 49].”
Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision (2020, background): “To cope with this limitation, [31] focuses on learning the global and local label distributions of images as the first task to regularize the model predictions in the target domain.”
Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID (2020, methodology): “Recently, some methods [39, 12, 4, 52, 60] incorporated the conception of self-paced learning into unsupervised learning tasks by starting the training process with the most confident pseudo labels.”
Adversarial Domain Adaptation With Prototype-Based Normalized Output Conditioner (2020, methodology): “For Synthia , we take the SYNTHIA-RAND-CITYSCAPES set as the source domain, which contains 9,400 synthetic images compatible with 16 annotated classes of Cityscapes [57].”