Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks

By Dong-Hyun Lee. 2013

4,728 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “We compare UDA with Pseudo-Label [36], an algorithm based on self-training, Virtual adversarial training (VAT) [44], an algorithm that generates adversarial Gaussian perturbations on input, Π -Model [35], which combines simple input augmentation with hidden state perturbations, Mean Teacher [58], wh”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “(15) FixerOnly can also be viewed as self-training (Lee, 2013) with the difference that we only add fixer outputs verified by the critic to the training data.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Works based on pseudo label [44, 37, 68, 1] are similar to self-training, but also suffer the same problem with consistency training, since they rely on a model being trained instead of a converged model with high accuracy to generate pseudo labels.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “An increasingly popular approach in deep learning consists in propagating information learned from labeled images to unlabeled ones [23, 39] while enforcing invariance to perturbations [44, 59, 65].”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Two additional important approaches for semisupervised learning, which have shown success both in the context of deep neural networks and other types of models are Pseudo-Labeling [19], where one imputes approximate classes on unlabeled data by making predictions from a model trained only on labeled”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “While previous methods based on bootstrapping have used pseudo-labels [16], cluster indices [17] or a handful of labels [18, 19, 20], we propose to directly bootstrap the representations.”
Challenges in representation learning: A report on three machine learning contests (2013, background): “Dong-Hyun Lee [8] finished second in the contest, having independently rediscovered entropy regularization [9].”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “This propagation can either be done with hard assignments of labels [41, 78, 79] or with a soft assignment [76].”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): ““Pseudo-Label” [28] does entropy minimization implicitly by constructing hard labels from high-conﬁdence predictions on unlabeled data and using these as training targets in a standard cross-entropy loss.”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “In the semi-supervised setting, we compare ELV-EST against several competitive semi-supervised text classification methods including Pseudo-Labeling [29], Self-Training [30], and Data Programming [13] which incorporates NL explanations to perform semi-supervised text classification.”
Revisiting Self-Training for Neural Sequence Generation (2019, background): “In the context of classiﬁcation, self-training can be understood from the view of entropy regularization (Lee, 2013), which favors a low-density separation between classes, a commonly assumed prior for semi-supervised learning (Chapelle & Zien, 2005).”
StarCoder: may the source be with you! (2023, methodology): “Pseudo-labels To improve the detection of key and password entities, we employed a pseudo-labeling technique as described by Lee (2013).”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, positive): “To fully exploit unsupervised data, we take a self-training framework similar to (Lee et al., 2013; Du et al., 2021).”
Semi-supervised Learning with Ladder Networks (2015, background): “Embedding (Weston et al. , 2012) 16.86 5.73 1.5 Transductive SVM (from Weston et al. , 2012) 16.81 5.38 1.40* MTC (Rifai et al. , 2011b) 12.03 3.64 0.81 Pseudo-label (Lee, 2013) 10.49 3.46 AtlasRBF (Pitelis et al. , 2014) 8.10 ( ± 0.95) 3.68 ( ± 0.12) 1.31 DGN (Kingma et al. , 2014) 3.33 ( 0.14) 2.4”
Learning with Pseudo-Ensembles (2014, methodology): “Thus, V x combines the KL-divergence penalty with an entropy penalty, which has been shown to perform well in a semi-supervised setting [7, 14].”