Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

By Zhedong Zheng, Yi Yang. 2020

584 citations recorded by the source corpus.

This paper focuses on the unsupervised domain adaptation of transferring the knowledge from the source domain to the target domain in the context of semantic segmentation. Existing approaches usually regard the pseudo label as the ground truth to fully exploit the unlabeled target-domain data. Yet the pseudo labels of the target-domain data are usually predicted by the model trained on the source domain. Thus, the generated labels inevitably contain the incorrect prediction due to the discrepancy between the training domain and the test domain, which could be transferred to the final adapted model and largely compromises the training process. To overcome the problem, this paper proposes to explicitly estimate the prediction uncertainty during training to rectify the pseudo label learning for unsupervised semantic segmentation adaptation. Given the input image, the model outputs the semantic segmentation prediction as well as the uncertainty of the prediction. Specifically, we model the uncertainty via the prediction variance and involve the uncertainty into the optimization objective. To verify the effectiveness of the proposed method, we evaluate the proposed method on two prevalent synthetic-to-real semantic segmentation benchmarks, i.e., GTA5 →\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow $$\end{document} Cityscapes and SYNTHIA →\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow $$\end{document} Cityscapes, as well as one cross-city benchmark, i.e., Cityscapes →\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow $$\end{document} Oxford RobotCar. We demonstrate through extensive experiments that the proposed approach (1) dynamically sets different confidence thresholds according to the prediction variance, (2) rectifies the learning from noisy pseudo labels, and (3) achieves significant improvements over the conventional pseudo label learning and yields competitive performance on all three benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Class-incremental Novel Class Discovery (2022, background): “Being meaningful, such source-free model adaptation has been explored in the related areas of domain adaptation [28,35].”
DACS: Domain Adaptation via Cross-domain Mixed Sampling (2020, methodology): “While existing works have proposed other improvements for how to correct erroneous pseudo-label generation arising due to the domain shift [17, 18, 22], we instead propose a change in the augmentation procedure, detailed in the next subsection.”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, methodology): “Hence, [76] adds confidence regularization terms to the network, while [73] explicitly estimates a prediction confidence map to reduce the side-effect of unreliable labels.”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, methodology): “RPLL (Zheng & Yang, 2021) and MRKLD+LRENT (Zou et al., 2019) are two commonly used pseudo-label generation methods, we reproduce their pseudo-label generations on our baseline to compare with our proposed pseudo labeling method.”
BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation (2021, methodology): “Following the common experimental setup for crossdomain semantic segmentation [50, 37, 38, 51, 52, 61, 66, 64], we evaluate our method on two synthetic-to-real scenarios, namely GTA5 to Cityscapes and SYNTHIA to Cityscapes.”
MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation (2022, methodology): “To increase the robustness of the self-training, consistency regularization [63, 67, 71] is often applied to ensure consistency over different data augmentations [1, 9, 18, 54], different crops [31, 38], multiple models [88, 93, 94], or domain-mixup [30–32, 72, 95].”
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling (2021, background): “An extra classifier is added to automate the threshold to deal with domain inconsistency in [41].”
Semi-supervised Left Atrium Segmentation with Mutual Consistency Training (2021, methodology): “Therefore, inspired by [20], we simplify the Monte Carlo dropout via introducing an auxiliary classifier, which reduces both training and inference costs.”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, unknown): “…pseudo-label drift, approaches such as confidence thresholding [48,99], pseudo-label prototypes [41,87,88], and consistency regularization [55, 57, 62] based on data augmentation [1, 10, 49, 51], different context [36,96], domain-mixup [21,30,41,63,96], or multiple models [86,94,95] have been used.”
Domain Consensus Clustering for Universal Domain Adaptation (2021, background): “Some recent works take advantage of adversarial training to promote the alignment in the input space [11, 23, 25, 20, 10] or feature space [58, 44, 6, 18, 38, 64, 43].”
Mutual consistency learning for semi-supervised medical image segmentation (2021, methodology): “In this paper, inspired by (Zheng and Yang, 2021), our model pre-deﬁned multiple sub-models before the model training, which estimates the model’s epistemic uncertainty in only one forward pass.”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, methodology): “In [50], threshold is modified by an extra classifier.”
In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning (2021, background): “Zheng & Yang (2020) show strong performance on the domain adaptive semantic segmentation task by leveraging uncertainty.”
Applications of Unsupervised Deep Transfer Learning to Intelligent Fault Diagnosis: A Survey and Comparative Study (2019, background): “For example, the prediction uncertainty is explicitly estimated during training to rectify the pseudo label learning for UDTL of semantic segmentation [192].”
Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation (2022, methodology): “Thus, we reproduce FADA [44] based on the modified ASPP when compared with ProDA and R-MRNet.”