Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning

By Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, Vicente Ordonez. AAAI Conference on Artificial Intelligence, 2020

377 citations recorded by the source corpus.

In this paper we revisit the idea of pseudo-labeling in the context of semi-supervised learning where a learning algorithm has access to a small set of labeled samples and a large set of unlabeled samples. Pseudo-labeling works by applying pseudo-labels to samples in the unlabeled set by using a model trained on combination of the labeled samples and any previously pseudo-labeled samples, and iteratively repeating this process in a self-training cycle. Current methods seem to have abandoned this approach in favor of consistency regularization methods that train models under a combination of different styles of self-supervised losses on the unlabeled samples and standard supervised losses on the labeled samples. We empirically demonstrate that pseudo-labeling can in fact be competitive with the state-of-the-art, while being more resilient to out-of-distribution samples in the unlabeled set. We identify two key factors that allow pseudo-labeling to achieve such remarkable results (1) applying curriculum learning principles and (2) avoiding concept drift by restarting model parameters before each self-training cycle. We obtain 94.91% accuracy on CIFAR-10 using only 4,000 labeled samples, and 68.87% top-1 accuracy on Imagenet-ILSVRC using only 10% of the labeled samples.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/39e0a6bceec1fe851128c47fedf35a2d4f22b332

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, methodology): “Two main branches of methods are proposed in recent years, namely consistency regularization [4, 21, 32, 39, 45, 51, 52] and entropy minimization [7, 33, 42, 53].”
Milking CowMask for Semi-Supervised Image Classification (2020, background): “…classiﬁcation approaches have been proposed over the last several years, including the use of auto-encoders (Wang et al., 2019; Rasmus et al., 2015), GANs (Salimans et al., 2016; Dai et al., 2017), curriculum learning (Cascante-Bonilla et al., 2020) and self-supervised learning (Zhai et al., 2019).”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, background): “Curriculum learning is introduced to adjust the threshold in [51, 21].”
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling (2021, background): “[42] introduced curriculum learning to self-training with a steadily increasing threshold and achieved near state-of-the-art results.”
Self-Training: A Survey (2022, positive): “Following this idea, [11] revisited the concept of pseudo-labeling by discussing the iterative process of assigning pseudo-labels to unlabeled data and emphasized the resilience of pseudo-labeling to out-of-distribution samples.”