Semi-supervised Learning with Deep Generative Models

By Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, Max Welling. 2014

2,940 citations recorded by the source corpus.

The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis. We revisit the approach to semi-supervised learning with generative models and develop new models that allow for effective generalisation from small labelled data sets to large unlabelled ones. Generative approaches have thus far been either inflexible, inefficient or non-scalable. We show that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “Similarly, VAEs [5] and Adversarial Autoencoders [9] were shown to learn representations in which class label is separated from other variations.”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Similarly, with the variational auto-encoding framework and reinforce algorithm, classic graphical models based SSL methods with target variable being latent can also take advantage of deep architectures [30, 39, 65].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Prior work has explored diverse self-supervision cues such as predicting the spatialcontext [12], colorization [71], equivariance to transformations [18, 44]; alongside unsupervised techniques such as clustering [6, 72], generative modelling [14, 32], and exemplar learning [15].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Finally, frameworks in semi-supervised learning also include graph-based methods [84, 73, 77, 33], methods that make use of latent variables as target variables [32, 42, 78] and methods based on low-density separation [21, 58, 15], which might provide complementary benefits to our method.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Sometimes generative models are also evaluated using linear classiﬁcation (Radford et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), or semi-supervised learning (Kingma et al., 2014; Narayanaswamy et al., 2017; Tschannen et al., 2018).”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, positive): “Representation learning saw early success using generative modeling (Kingma et al., 2014), but likelihood-based models have yet to generalize to more complex stimuli.”
S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Many of the initial results on semi-supervised learning with deep neural networks were based on generative models such as denoising autoencoders [31], variational autoencoders [15] and generative adversarial networks [27, 33].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In the semi-supervised setting [55, 56], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [19, 20, 57, 58, 59, 60, 61, 62].”
Learning to generate chairs with convolutional neural networks (2014, critical): “In contrast to our work, these approaches are typically restricted to small models and images, and they often require an expensive inference procedure – both during training and for generating images.”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “To list a few, they have been used to generate high-fidelity images [26, 6], synthesize realistic speech and music fragments [58], improve the performance of semi-supervised learning [28, 10], detect adversarial examples and other anomalous data [54], imitation learning [22], and explore promising s”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “BIVA can be easily extended for semi-supervised classification by adding a categorical variable y to represent the class, as done in [22].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “Our method also outperforms some meth-ods that are speciﬁcally designed for semi-supervised learning, such as DGN [17], virtual adversarial [30] and auxiliary deep generative model [29].”
Improved Techniques for Training GANs (2016, methodology): “We use this data set to study semi-supervised learning, as well as to examine the visual quality of samples that can be achieved.”
Variational Inference with Normalizing Flows (2015, background): “…now lies at the core of large-scale topic models of text (Hoffman et al., 2013), provides the state-of-the-art in semi-supervised classiﬁcation (Kingma et al., 2014), drives the models that currently produce the most realistic generative models of images (Gre-gor et al., 2014; 2015; Rezende…”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “In a conditional setting [36], latent variables which are independent from the conditioning lead to disentangled representations [28, 66, 45, 57].”
Score-based Generative Modeling in Latent Space (2021, background): “Furthermore, like other deep generative models, it can potentially be used also for non-generative tasks such as semi-supervised and representation learning [101, 102, 103].”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “…al., 2016a) generation, distribution alignment across domains (Zhu et al., 2017; Liu et al., 2017; Tzeng et al., 2017) and semi-supervised learning (Kingma et al., 2014; Izmailov et al., 2020) to 3D point cloud generation (Yang et al., 2019), light-transport simulation (Müller et al., 2019),…”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “While the quality of results from generative adversarial networks (GAN) [8], variational autoencoders (VAE) [15], autoregressive models [28, 29], and likelihood-based models [6, 14] have seen rapid improvement recently [12, 9, 27, 20, 4, 13], the automatic evaluation of these results continues to be”