Learning Discrete Representations via Information Maximizing Self-Augmented Training

By Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama. 2017

495 citations recorded by the source corpus.

Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invari-ance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, background): “Other approaches, such as those presented by Hu et al. (2017) and Ji et al. (2018), can be similarly subsumed under the same objective.”
Learning deep representations by mutual information estimation and maximization (2018, background): “It has been shown in the case of discrete MI that data augmentations and other transformations can be used to avoid degenerate solutions (Hu et al., 2017).”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “Optimising for function outputs to be persistent through spatio-temporal or non-material distortion is an idea shared by IIC with several works, including exemplars [18], IMSAT [28], proximity prediction [30], the denoising objective of Tagger [22], temporal slowness constraints [55], and optimising”
Stacked Capsule Autoencoders (2019, methodology): “To test this hypothesis, (Haeusser et al., 2018) 53.49 20.8 12.5 AE (Bengio et al., 2007) § 81.2 31.4 - GAN (Radford et al., 2016) § 82.8 31.5 - IMSAT (Hu et al., 2017) † , ∇ 98.4 (0.4) 45.6 (0.8) 57.3 (3.9) IIC (Ji et al., 2018) § , † we train SCAE on MNIST , SVHN 8 and CIFAR 10 and try to assign c”
On Variational Bounds of Mutual Information (2019, background): “…x ∼ p ( x ) , the goal is to learn a stochastic representation of the data p θ ( y | x ) that has maximal MI with X subject to constraints on the mapping (e.g. Bell & Sejnowski, 1995; Krause et al., 2010; Hu et al., 2017; van den Oord et al., 2018; Hjelm et al., 2018; Alemi et al., 2017).”
Consistency Regularization for Generative Adversarial Networks (2019, methodology): “In this paper, we examine a technique called consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2016; Zhai et al., 2019; Xie et al., 2019; Hu et al., 2017) in contrast to gradient-based regularizers.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “To better guide the learning process, previous works [298], [299] modify the mutual information regularization by substituting a reference class-ratio distribution in place of U .”
Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer (2020, methodology): “feature representations and outputs of the classifier, since information maximization [26], [27] can encourage the classifier to assign disparate one-hot outputs to different target feature representations.”
Joint Optimization Framework for Learning with Noisy Labels (2018, methodology): “Therefore, we introduce the KL-divergence from ¯ s ( θ , X ) to p as a cost function as follows: This approach follows [10].”
SpectralNet: Spectral Clustering using Deep Neural Networks (2018, methodology): “IMSAT (Hu et al., 2017) is based on data augmentation, where the net is trained to maximize the mutual information between inputs and predicted clusters, while regularizing the net so that the cluster assignment of original data points will be consistent with the assignment of augmented points.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Deep clustering models learn discriminative representations by grouping similar instances from the same cluster together [138], [139], [140], [142], [144], [170], [171], [182], [183], [184], [185], [186], [187].”
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information (2020, background): “MI maximization has been studied extensively in various tasks, e.g. , representation learning (Hjelm et al., 2018; Hu et al., 2017), generative models (Chen et al., 2016), information distillation (Ahn et al., 2019), and reinforcement learning (Florensa et al., 2017).”
Clustering with Deep Learning: Taxonomy and New Methods (2018, methodology): “In addition to the mentioned methods, multiple others exist (Premachandran and Yuille, 2016; Harchaoui et al., 2017; Zheng et al., 2016; Chen et al., 2017; Lukic et al., 2016; Wang et al., 2016; Chen, 2015; Hu et al., 2017).”
Associative Deep Clustering: Training a Classification Network with No Labels (2018, methodology): “In IMSAT this mutual information is represented by the difference between the marginal and conditional distribution of those values.”
On variational lower bounds of mutual information (2018, background): “…x ∼ p ( x ) and are interested in learning a stochastic mapping p θ ( y | x ) that captures as much information as possible about x subject to constraints on the mapping (Bell & Sejnowski, 1995; Krause et al., 2010; Hu et al., 2017; Hjelm et al., 2018; Alemi et al., 2017; McAllester, 2018).”
Learning To Classify Images Without Labels (2020, methodology): “In contrast to prior work [6,24,21], we did not have to perform any dataset specific fine-tuning.”
Greedy InfoMax for Biologically Plausible Self-Supervised Representation Learning (2019, background): “Thus, the InfoNCE loss can successfully enforce each module to preserve the information of its inputs, while providing the necessary regularization [Hu et al., 2017, Krause et al., 2010] for circumventing degenerate solutions.”