Ladder Variational Autoencoders

By Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther. 2016

1,005 citations recorded by the source corpus.

Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “In the next section we will introduce the Bidirectional-Inference Variational Autoencoder, that manages to avoid these issues by extending the LVAE in 2 ways: (i) adding a deterministic top-down path in the generative model and (ii) deﬁning a factorization of the latent variables z i at each level o”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “Some VAE variants (Sønderby et al., 2016; Kingma et al., 2016) however mix the p -parameters implicitly in forming q , by using the likelihood gradient to iteratively form the approximate posterior.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Consistent with ﬁndings in Bowman et al. (2015) and Kaae Sønderby et al. (2016), we found that stochastic optimization with the unmodiﬁed lower bound objective often gets stuck in an undesirable stable equilibrium.”
Score-based Generative Modeling in Latent Space (2021, background): “[90] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “One particularly elegant conditioning structure is the top-down VAE , introduced in Sønderby et al. (2016).”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “In deep hierarchical VAEs [5, 9, 4, 42, 43], to increase the expressiveness of both the approximate posterior and prior, the latent variables are partitioned into disjoint groups, zzz = { zzz 1 ,zzz 1 , . . . ,zzz L } , where L is the number of groups.”
Auxiliary Deep Generative Models (2016, methodology): “The methods are not directly comparable, except for the Ladder VAE (LVAE) (Sønderby et al., 2016), since the training is performed differently.”
PixelVAE: A Latent Variable Model for Natural Images (2016, background): “3 LATENT REPRESENTATIONS On MNIST, given a sufficiently high-dimensional latent space, VAEs have already been shown to learn representations in which digits are well-separated (Sønderby et al., 2016).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “• We vary the parameter β in { 1 , 2 , 4 } , and use linear deterministic warm-up (Bowman et al., 2015; Sønderby et al., 2016) over the ﬁrst { 0 , 10000 , 50000 } training steps.”
Learning Disentangled Representations with Semi-Supervised Deep Generative Models (2017, background): “We note however, that from the perspective of achieving the lowest possible classification error, one could adopt any number of alternate factorisations [24] and innovations in neural-network architectures [27, 33].”
Bayesian Compression for Deep Learning (2017, background): “We also found that constraining the standard deviations as described at Louizos and Welling [2017] and “warm-up" [Sønderby et al., 2016] helps in avoiding bad local optima of the variational objective.”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “| Implicitly encoding a hierarchy into p θ ( z 1 | z 2 , . . . 15 z L ) rather than explicitly as by PixelVAE and LVAE avoids the difﬁculties described by [ ] involved with training hierarchical models with more than two levels of latent variables.”
A causal framework for explaining the predictions of black-box sequence-to-sequence models (2017, methodology): “VAEs have been successfully used with fixed dimensional inputs such as images (Rezende and Mohamed, 2015; Sønderby et al., 2016) and recently also adapted to generating sentences from continuous representations (Bowman et al.”
Discrete Variational Autoencoders (2016, methodology): “We use warm-up with strength 20 for 5 epochs, and additional warm-up of strength 2 on the RBM alone for 20 epochs (Raiko et al., 2007; Bowman et al., 2016; Sønderby et al., 2016).”
VAE with a VampPrior (2017, background): “Last but not least, it would be interesting to utilize a normalizing flow [32], the hierarchical variational inference [31], ladder networks [36] or adversarial training [28] within the VampPrior VAE.”
A Neural Representation of Sketch Drawings (2017, background): “Neural Network-based approaches have been developed for generative models of images, although the majority of neural network-related research on image generation deal with pixel images [5, 10, 12, 14, 19, 24].”
Sylvester Normalizing Flows for Variational Inference (2018, neutral): “We use annealing to optimize the lower bound, where the prefactor of the KL divergence is linearly increased from 0 to 1 during 100 epochs as suggested by Bowman et al. [2015] and Sønderby et al. [2016].”