Auxiliary Deep Generative Models

By Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, Ole Winther. 2016

467 citations recorded by the source corpus.

Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged but make the variational distribution more expressive. Inspired by the structure of the auxiliary variable we also propose a model with two stochastic layers and skip connections. Our findings suggest that more expressive and properly specified deep generative models converge faster with better results. We show state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “Similarly, with the variational auto-encoding framework and reinforce algorithm, classic graphical models based SSL methods with target variable being latent can also take advantage of deep architectures [30, 39, 65].”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, methodology): “For semi-supervised learning with neural networks, a common approach, especially in the image domain, is to train a generative model whose latent features may be used as features for classiﬁ-cation (e.g. (Hinton et al., 2006; Maaløe et al., 2016)).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Finally, frameworks in semi-supervised learning also include graph-based methods [84, 73, 77, 33], methods that make use of latent variables as target variables [32, 42, 78] and methods based on low-density separation [21, 58, 15], which might provide complementary benefits to our method.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “R | Agakov and Barber [1], Salimans et al. [61], Ranganath et al. [57], Maaløe et al. [46] develop an auxiliary variable variational bound (1)), however, the density of the EIM q ( z x ) is intractable. where r ( | z, x ) is a variational distribution meant to model q ( | z, x ) , and the identity…”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “The inspiration for the bidirectional inference network of BIVA comes from the work on Auxiliary VAEs (AVAE) by [37, 31].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “Our method also outperforms some meth-ods that are speciﬁcally designed for semi-supervised learning, such as DGN [17], virtual adversarial [30] and auxiliary deep generative model [29].”
Improved Techniques for Training GANs (2016, methodology): “DGN[21] 333 ± 14 VirtualAdversarial[22] 212 CatGAN[14] 191 ± 10 SkipDeepGenerativeModel[23] 132 ± 7 Laddernetwork[24] 106 ± 37 AuxiliaryDeepGenerativeModel[23] 96 When presented with 50% real and 50% fake data generated by our best CIFAR-10 model, MTurk users correctly categorized 78 .”
Density estimation using Real NVP (2016, background): “Still, the approximation in the inference process limits its ability to learn high dimensional deep representations, motivating recent work in improving approximate inference [42, 48, 55, 63, 10, 59, 34].”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…[1, 2] is dedicated to the statistical challenges, such as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20,…”
Unsupervised Image-to-Image Translation Networks (2017, neutral): “By improving the variational approximation, better image generation results were achieved (Maaløe et al., 2016; Kingma et al., 2016).”
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, background): “Aux-iliary Variable VAEs (Maaløe et al., 2016) add auxiliary variables to the posterior to make it more ﬂexible.”
Adversarial Autoencoders (2015, methodology): “Recently, generative models have become one of the most popular approaches for semi-supervised learning as they can disentangle the class label information from many other latent factors of variation in a principled way [Kingma et al., 2014, Maaløe et al., 2016].”
Adversarially Learned Inference (2016, neutral): “SDGM (Maaløe et al. , 2016) 16 .”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “SWWAE [43] 23.56 Skip Generative Model [24] 16.30 Ladder networks, Γ model [30] 20.40 CatGAN [33] 19.58 GAN with feature matching [32] 8.11 18.63 model [21] 5 Table 5 shows the performance of VAT and contemporary methods implemented with moderate image data augmentation (translation and horizontal…”
Temporal Ensembling for Semi-Supervised Learning (2016, methodology): “Generative Adversarial Networks (GAN) have been recently used for semi-supervised learning with promising results [8, 18, 16].”
Neural Photo Editing with Introspective Adversarial Networks (2016, neutral): “SDGM (Maaløe et al., 2016) 16 .”
Ladder Variational Autoencoders (2016, methodology): “Further the LVAE is orthogonal to other methods for improving the inference distribution such as Normalizing ﬂows [17], Variational Gaussian Process [20] or Auxiliary Deep generative models [12] and combining with these might provide further improvements.”
Causal Effect Inference with Deep Latent-Variable Models (2017, neutral): “…work, we plan to employ the expanding set of tools available for latent variables models (e.g. Kingma et al. [28], Tran et al. [51], Maaløe et al. [35], Ranganath et al. [44]), as well as to further explore connections between method of moments approaches such as Anandkumar et al. [5] with the…”