Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

By Eric Arazo, Diego Ortego, Paul Albert, Noel Edward O'Connor, Kevin McGuinness. 2019

1,104 citations recorded by the source corpus.

Semi-supervised learning, i.e. jointly learning from labeled and unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Source code is available at https://git.io/fjQsC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “However, recent SSL algorithms have used pseudo-labeling as a part of their pipeline to produce better results [1, 32].”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “[2], other semi-supervised self-training in semantic segmentation [30, 31, 32, 33, 34], and other approaches based on pseudo label such as [4, 35, 36, 3].”
Debiased Pseudo Labeling in Self-Training (2022, background): “However, this paradigm still suffers from the problem of confirmation bias (Arazo et al., 2020), where the learner struggles to correct its own mistakes when learning from inaccurate pseudo labels.”
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, background): “Nevertheless, different unlabeled images cannot be equally easy [34, 44, 50] and the corresponding pseudo labels cannot be equally reliable, leading to severe confirmation bias [2] and potential performance degradation when iteratively optimizing the model with those ill-posed pseudo labels.”
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases (2020, background): “Self-supervised Self-supervised models exhibit the same tendency [2].”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, methodology): “We study the original version of fairness objective as in (Arazo et al., 2020).”
Semi-Automated Data Labeling (2020, methodology): “Reﬁnements of the basic algorithm include the use of conﬁdence thresholds and soft labels Arazo et al. (2020).”
Decoupled Mixup for Data-efficient Learning (2022, neutral): “However, if the reliability of pseudo labels is not considered, the generated samples with false pseudo labels (FPLs) can even mislead the model and aggravate confirmation bias (Arazo et al., 2020), as shown in Figure 3 left.”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “The effectiveness of this iterative training for a single model has been validated on tasks in vision [25], natural language processing [18], and ASR [26, 5].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “…to all the unlabeled training data at once, addressing these challenges is nontrivial as directly applying many existing SSL and UL methods could not guarantee good generalization performance, e.g., pseudo labels may suffer the conﬁrmation bias problem [241] when classifying unseen unlabeled data.”
Recent advances and clinical applications of deep learning in medical image analysis (2021, methodology): “The naïve pseudo-labeling process can be combined with MixUp augmentation (Zhang et al., 2018a) to further improve SSL model’s performance (Arazo et al., 2020).”
Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation (2022, background): “Intuitively, its success lies in that the model is more likely to produce high-quality prediction on x, while x is more effective for our model to learn, since the strong perturbations introduce additional information as well as mitigate confirmation bias [2].”
An Overview of Deep Semi-Supervised Learning (2020, background): “[2] showed that a naive pseudo-labeling overﬁts to incorrect pseudo-labels due to the so-called conﬁrmation bias, and demonstrate that MixUp [178] and setting a minimum number of labeled samples per mini-batch are eﬀective regularization techniques for reducing this bias.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, background): “The problem of confirmation bias in pseudo-labeling [239] can make students learn from inaccurate data if the pseudo labels are inaccurate.”