In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

By Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Singh Rawat, Mubarak Shah. 2021

655 citations recorded by the source corpus.

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, background): “Since the examples with high uncertainty are difficult to estimate the true label, we treat them as the unlabeled set instead to prevent false supervision [30].”
Unsupervised Prompt Learning for Vision-Language Models (2022, background): “Self-training (Scudder 1965; Yarowsky 1995; Riloff 1996) is a simple semi-supervised learning approach.”
Mutual consistency learning for semi-supervised medical image segmentation (2021, methodology): “The second category consists of several entropy-minimization methods (Lee et al., 2013; Rizve et al., 2021; Pham et al., 2021), which are based on the cluster assumption, i.e., the cluster of each class should be compact and thus of low entropy .”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, background): “Pseudo labeling [11, 12, 13, 14] and consistency regularization [15, 16, 17] are two popular paradigms designed for modern SSL.”
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels (2022, methodology): “UPS [35], a method based on FixMatch [38], takes model uncertainty and data uncertainty into consideration.”
Debiased Pseudo Labeling in Self-Training (2022, methodology): “Different self-training methods design different pseudo labeling function f̂ (Lee, 2013; Xu et al., 2021; Rizve et al., 2021).”
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling (2021, background): “A possible reason is that the lower threshold prevents noisy training caused by incorrect pseudo labels at the early stage [36].”
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2021, methodology): “Moreover, given the many existing labelled datasets, another area of opportunity is developing methods that leverage characteristics of labelled datasets to generate labels for unlabeled datasets such as transfer learning or pseudo-label method [292].”