Fully Convolutional Adaptation Networks for Semantic Segmentation

By Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, Tao Mei. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

369 citations recorded by the source corpus.

The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets. Nevertheless, collecting expert labeled datasets especially with pixel-level annotations is an extremely expensive process. An appealing alternative is to render synthetic data (e.g., computer games) and generate ground truth automatically. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. In this paper, we facilitate this issue from the perspectives of both visual appearance-level and representation-level domain adaptation. The former adapts source-domain images to appear as if drawn from the "style" in the target domain and the latter attempts to learn domain-invariant representations. Specifically, we present Fully Convolutional Adaptation Networks (FCAN), a novel deep architecture for semantic segmentation which combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN). AAN learns a transformation from one domain to the other in the pixel space and RAN is optimized in an adversarial learning manner to maximally fool the domain discriminator with the learnt source and target representations. Extensive experiments are conducted on the transfer from GTA5 (game videos) to Cityscapes (urban street scenes) on semantic segmentation and our proposal achieves superior results when comparing to state-of-the-art unsupervised adaptation techniques. More remarkably, we obtain a new record: mIoU of 47.5% on BDDS (drive-cam videos) in an unsupervised setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7c30ee6470500f47eefacfa0560a8918e229138c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, background): “Besides, [16, 52] try to eliminate the domain gap on pixel-level by translating images.”
Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation (2019, background): “A major class of adaptation approaches, including [5,12,13,19,28,31,34,36,43,46], involves adversarial confusion or feature alignment between domains.”
Confidence Regularized Self-Training (2019, neutral): “2 FCAN [67] - - - - - - - - - - - - - - - - - - - 46.”
Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation (2019, methodology): “Some methods ﬁrst trained a DCNN in source domain, and then transformed the target images into source-like ones, such that can be tested using the pre-trained source model (Russo et al. 2017; Zhang et al. 2018b; Chen et al. 2018).”
Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes (2019, background): “Following these approaches, many works addressed the source to target domain shift problem with various techniques, such as cycle consistency [11], GANs [29], output space alignment [35], distillation loss [4], classbalanced self-training [46], conservative loss [45], geometrical guidance [3] and ad”
DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation (2018, background): “Generative Adversarial Networks (GANs) [35], consisting of a generator and a discriminator optimized to compete with each other, are one of the most popular deep generative models for image synthesis [36,37,33].”
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “A major class of UDA works employs adversarial domain adaptation to implicitly align the source and target domains at the level of pixels and/or features [7, 15, 26, 41, 43, 44, 46, 48, 60].”
Domain Adaptation for Structured Output via Discriminative Patch Representations (2019, background): “, scene layout) across domains; 2) pixel-level adaptation synthesizes target samples [15, 27, 43, 47] to reduce the domain gap during training the segmentation model; 3) pseudo-label re-training [34, 50] generates pseudo ground truth of target images to finetune the model trained on the source domai”
A Review of Single-Source Deep Unsupervised Visual Domain Adaptation (2020, background): “homogeneity homogeneous DA dS = dT source and target data are observed in the same space [11, 23, 36, 44–106, 113–151] heterogeneous DA dS 6= dT source and target data are observed in different spaces [107–112]”
Adversarial Learning and Self-Teaching Techniques for Domain Adaptation in Semantic Segmentation (2019, background): “…problem with various techniques, including GANs [29], [45], cycle consistency [11], [46], output space alignment [47], [48], distillation loss [17], [49], class-balanced self-training [50], conservative loss [51], geometrical guidance [52], adaptation networks [53] and entropy minimization [54].”
IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks (2019, background): “Image segmentation: Currently, most state-of-the-art models [8, 9, 10] for semantic segmentation stem from a fully-convolutional design, which is first introduced by the fully convolutional network (FCN) [11].”
Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation (2020, background): “Importantly, the combination between image and feature alignment should be a synergistic merge to exploit their mutual interactions and benefits, which has not been tapped in previous works [18], [19].”
Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection (2019, background): “As unsupervised domain adaptation attracted considerable interest with its effectiveness, recent works [17, 47, 6, 5, 38, 43, 19, 48] have been attempted to address the generalization issue in the semantic segmentation task.”
Exploring Object Relation in Mean Teacher for Cross-Domain Detection (2019, background): “, object detection [4, 35] and semantic segmentation [5, 16, 53].”
CE-Net: Context Encoder Network for 2D Medical Image Segmentation (2019, methodology): “2) Testing Phase: To improve the robustness of medical image segmentation method, we also adopt test augmentation strategy, as that in [60] and [61], including image horizontal flip, vertical flip and diagonal flip (equal to predicting each image 8 times).”
Transferrable Prototypical Networks for Unsupervised Domain Adaptation (2019, background): “Another branch of unsupervised domain adaptation in DCNNs is to exploit the domain confusion by learning a domain discriminator [4, 14, 29, 30, 35].”
DACS: Domain Adaptation via Cross-domain Mixed Sampling (2020, methodology): “For all the experiments in this paper, we adopt the widely used [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22] DeepLab-v2 framework [41] with a ResNet101 backbone [42] as our model.”
BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation (2021, methodology): “Following the common experimental setup for crossdomain semantic segmentation [50, 37, 38, 51, 52, 61, 66, 64], we evaluate our method on two synthetic-to-real scenarios, namely GTA5 to Cityscapes and SYNTHIA to Cityscapes.”