Entropy Regularization

By Yves Grandvalet, Yoshua Bengio. 2006

108 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Optimization & Training

Citations in the corpus

Discriminative Unsupervised Feature Learning with Convolutional Neural Networks (2014, neutral): “This work is also loosely related to the use of unlabeled data for regularizing supervised algorithms, for example self-training [15] or entropy regularization [16].”
Dropout Training as Adaptive Regularization (2013, background): “Our semi-supervised approach is based on a different intuition: we’d like to set weights to make conﬁdent predictions on unlabeled data as well as the labeled data, an intuition shared by entropy regularization [24] and transductive SVMs [25].”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks (2014, unknown): “This work is also loosely related to the use of unlabeled data for regularizing supervised algorithms, for example self-training [19], entropy regularization [20] or Discriminative and Shareable Feature Learning [21].”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “For Entropy Regularization (Grandvalet and Bengio 2006), it is the same as Pseudo Labeling.”
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations. (2020, methodology): “…the computational efﬁciency of S4L, it shows that S4L can be combined in a stage-wise approach with other semi-supervised methods such as Virtual Adversarial Training [36], Entropy Regularization [19], and Pseudo-Label [31] to improve the ﬁnal accuracy of their model (see follow-up work [49, 26]).”
Feature Noising for Log-Linear Structured Prediction (2013, background): “(2013) showed that semi-supervised dropout training for logistic regression captures a similar intuition to techniques such as entropy regularization (Grandvalet and Bengio, 2005) and transductive SVMs (Joachims, 1999), which encourage confident predictions on the unlabeled data.”
Training with Confusion for Fine-Grained Visual Classification (2017, background): “Regularizing Entropy: Regularization methods that penalize minimum entropy predictions have been explored in the context of semi-supervised learning [9].”
Asymptotically Optimal Regularization in Smooth Parametric Models (2009, methodology): “One method of using the unlabeled data is to use entropy regularization [18], which minimizes the entropy of a learned distribution pθ(y | x) as measured on unlabeled points.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “…task-specific classification model may encourage a more deterministic de-3 cision boundary, which shares the same spirits with entropy minimization (Grandvalet and Bengio, 2006) or self-training (Lee et al., 2013), and 2) classification tasks benefit from the inductive bias in the architecture.”
Understanding Self-Training for Gradual Domain Adaptation (2020, background): “, 2018), and is related to entropy minimization (Grandvalet and Bengio, 2005).”
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (2021, methodology): “For example, [12, 13, 14] train a model by adding an unsupervised regularization term to a supervised cross-entropy loss.”
Decoupling "when to update" from "how to update" (2017, methodology): “This method is inspired by a technique called minimum entropy regularization, detailed in [16, 17].”