Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

By Jost Tobias Springenberg. 2015

777 citations recorded by the source corpus.

In this paper we present a method for learning a discriminative classifier from unlabeled or partially labeled data. Our approach is based on an objective function that trades-off mutual information between observed examples and their predicted categorical class distribution, against robustness of the classifier to an adversarial generative model. The resulting algorithm can either be interpreted as a natural generalization of the generative adversarial networks (GAN) framework or as an extension of the regularized information maximization (RIM) framework to robust classification against an optimal adversary. We empirically evaluate our method - which we dub categorical generative adversarial networks (or CatGAN) - on synthetic data as well as on challenging image classification tasks, demonstrating the robustness of the learned classifiers. We further qualitatively assess the fidelity of samples generated by the adversarial generator that is learned alongside the discriminative classifier, and identify links between the CatGAN objective and discriminative clustering algorithms (such as RIM).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, neutral): “Other possible extensions to this work include: learning hierarchical latent representations, improving semi-supervised learning with better codes [31], and using InfoGAN as a high-dimensional data discovery tool.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “12 % C AT GAN [51] 1 .”
Improved Techniques for Training GANs (2016, methodology): “Like many deep generative models, GANs have previously been applied to semi-supervised learning [13, 14], and our work can be seen as a continuation and reﬁnement of this effort.”
Generative adversarial networks (2014, neutral): “Springenberg (2015) Generator tries to fool discriminator into assigning a label (low label entropy) Discriminator tries to leave generated samples as uncategorized (high entropy) while real samples are categorized (low entropy) Optional supervision on labels can be provided”
Adversarial Autoencoders (2015, methodology): “On the MNIST dataset with 100 and 1000 labels, the performance of AAEs is signiﬁcantly better than VAEs, on par with VAT [Miyato et al., 2015] and CatGAN [Springenberg, 2015], but is outperformed by the Ladder networks [Rasmus et al., 2015] and the ADGM [Maaløe et al., 2016].”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “One shortcoming in the methods based on the generative model is that they are difﬁcult to interpret.”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “GANs have been successfully applied to semi-supervised learning at least since the introduction of CatGANs (Springenberg, 2015).”
Semi-Supervised Learning with Generative Adversarial Networks (2016, background): “In Radford et al. (2015), the features learned by D are reused in classifiers.”
Learning to Compose Domain-Specific Transformations for Data Augmentation (2017, methodology): “Standard unsupervised GANs can be used to generate additional out-of-class data points that can then augment labeled training sets [25, 29].”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, neutral): “There are several approaches to semi-supervised learning (Berthelot et al., 2019; Lee, 2013; Kingma et al., 2014; Kuo et al., 2019; Miyato et al., 2018; Oliver et al., 2018; Sohn et al., 2020; Springenberg, 2015; Tarvainen & Valpola, 2017) which involve balancing a supervised loss `s applied to the ”
High-Fidelity Image Generation With Fewer Labels (2019, background): “Similarly, Springenberg (2016) extends the standard GAN objective to K classes.”
Data-Free Learning of Student Networks (2019, neutral): “It is worth noticing that the combination of one-hot loss and information entropy is essential for training the generator, which is also utilized in some previous works [25, 10].”