Unsupervised Classifiers, Mutual Information and 'Phantom Targets'

By John Scott Bridle, A.J.R. Heading, David John Cameron MacKay. 1991

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “More formally, the minimax game is given by the following expression: min G max D V (D,G) = Ex∼Pdata [logD(x)] + Ez∼noise[log (1−D(G(z)))] (1)”
On Mutual Information Maximization for Representation Learning (2019, background): “Already Linsker (1988) talks in his seminal work about constraints, while a manifestation of the problem in clustering approaches using MI criteria has been brought up by Bridle et al. (1992) and subsequently addressed using regularization by Krause et al.”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, background): “…as already discussed, the idea of minimizing entropy of a classiﬁer on unlabeled data has been considered several times already in the literature (Bridle et al., 1992; Grandvalet & Bengio, 2005; Krause et al., 2010), and our objective function falls back to the regularized information…”
Third-Person Imitation Learning (2017, background): “(See Bridle et al. (1992); Barber & Agakov (2005); Krause et al.”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, methodology): “A P ROOF OF E QUATION 2 The proof here follows closely Bridle et al. (1992).”
Learning Discrete Representations via Information Maximizing Self-Augmented Training (2017, methodology): “We then combine the SAT with the Regularized Information Maximization (RIM) for clustering (Gomes et al., 2010; Bridle et al., 1991), and arrive at our Information Maximizing Self-Augmented Training (IMSAT), an information-theoretic method for learning discrete representations using deep neural netw”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, methodology): “An intuitive approach to leverage unlabeled data is to maximize the mutual information between the model’s input and output on unlabeled data [25]: I(y, x) = H(Eu [pm(y|u)])−Eu [H(pm(y|u))] , (5) whereEu is the expectation overDU andH(·) denotes entropy.”
Discriminative Clustering by Regularized Information Maximization (2010, background): “[4] in order to learn probabilistic classifiers that are appropriate for multi-class discriminative clustering, as explained in Section 2.”
ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring (2020, methodology): “One way to formalize this intuition, first proposed by Bridle et al. (1992), is to maximize the mutual information between the model’s input and output for unlabeled data.”
Discovering Predictable Classifications (1993, methodology): “This novel method is easy to implement { it achieves an e ect similar to the one of the recent entropy-based method by Bridle and MacKay (1992).2.2 AUTO-ENCODERSWith pattern p and classi er Tl a reconstructor module Al (another back-prop network) receives yp;l as aninput.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Another favorable clustering-based regularization is mutual information maximization, which tries to maximize the mutual information [286], [287] between inputs and the discrete labels as follows,”