Semi-supervised Learning by Entropy Minimization

By Yves Grandvalet, Yoshua Bengio. 2004

2,520 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Since regularizing the predictions to have low entropy has been shown to be beneficial [18, 6], we sharpen predictions when computing the target distribution on unlabeled examples by using a low Softmax temperature ⌧ .”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, background): “Entropy Minimization (EM) In addition to supervised cross-entropy loss, we also minimize the conditional entropy of the estimated class probabilities (Grandvalet and Bengio 2004; Miyato et al. 2018).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Finally, frameworks in semi-supervised learning also include graph-based methods [84, 73, 77, 33], methods that make use of latent variables as target variables [32, 42, 78] and methods based on low-density separation [21, 58, 15], which might provide complementary benefits to our method.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “One class of semi-supervised techniques consists in propagating the knowledge learned from the subset of labeled examples to unlabeled ones [23, 39], and learning from this new information.”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Two additional important approaches for semisupervised learning, which have shown success both in the context of deep neural networks and other types of models are Pseudo-Labeling [19], where one imputes approximate classes on unlabeled data by making predictions from a model trained only on labeled”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, neutral): “[37, 38, 30] ResNet-50 - - 47.0 83.4 UDA (w. RandAug) [14] ResNet-50 - 68.8 - 88.5 FixMatch (w. RandAug) [15] ResNet-50 - 71.5 - 89.1 S4L (Rot+VAT+Entropy Min.)”
Challenges in representation learning: A report on three machine learning contests (2013, background): “Dong-Hyun Lee [8] ﬁnished second in the contest, having independently rediscovered entropy regularization [9].”
Classifier-Free Diffusion Guidance (2022, background): “For example, Grandvalet & Bengio (2004) ﬁnd that discriminative models generally outperform implicit classiﬁers derived from generative models, even in artiﬁcial cases where the speciﬁcation of those generative models exactly matches the data distribution.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “Similarly, as mentioned above, pseudo-labeling results in a form of entropy minimization [16] which has been used as a component for many powerful SSL techniques [28].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “PIRL performs at least as well as S4L [84] and better than VAT [22], which are both methods specifically designed for semi-supervised learning.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “This is done explicitly in [18] with a loss term which minimizes the entropy of pmodel(y | x; θ) for unlabeled data x.”
Revisiting Self-Training for Neural Sequence Generation (2019, background): “1 re-2 veals the connection between self-training and entropy regularization (Grandvalet & Bengio, 2005).”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, background): “…the idea of minimizing entropy of a classiﬁer on unlabeled data has been considered several times already in the literature (Bridle et al., 1992; Grandvalet & Bengio, 2005; Krause et al., 2010), and our objective function falls back to the regularized information maximization from Krause et al.…”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “Also, In SVHN and CIFAR-10 experiments, we adopted conditional entropy of p ( y x, θ ) as a additional cost, used in [12] and similar one used in [31]: The conditional entropy minimization forces the model p ( y x, θ ) to exaggerate the prediction on each data point.”
Colorization as a Proxy Task for Visual Understanding (2017, background): “Semi-supervised learning, jointly training a supervised and an unsupervised loss, offers a middle ground [7, 34].”
Toward Controlled Generation of Text (2017, methodology): “To alleviate the issue of noisy data and ensure robustness of model optimization, we incorporate a minimum entropy regularization term (Grandvalet et al., 2004; Reed et al., 2014).”
Self-Training for Class-Incremental Semantic Segmentation (2020, background): “Recently, it has achieved signiﬁcant success on semisupervised learning [48], [51] and domain adaptation [13], [52].”
Training Deep Neural Networks on Noisy Labels with Bootstrapping (2014, background): “Minimum entropy regularization, proposed in (Grandvalet & Bengio, 2005; 2006), performs semi-supervised learning by augmenting cross-entropy loss with a term encouraging the classiﬁer to make predictions with high conﬁdence on the unlabeled examples 1 .”