How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

By Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther. 2016

142 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Consistent with ﬁndings in Bowman et al. (2015) and Kaae Sønderby et al. (2016), we found that stochastic optimization with the unmodiﬁed lower bound objective often gets stuck in an undesirable stable equilibrium.”
Variational Lossy Autoencoder (2016, positive): “In particular, the receptive ﬁeld window can be a small rectangle adjacent to a pixel x i and in this case long-range dependency will be encoded in the latent code z .”
Auxiliary Deep Generative Models (2016, methodology): “The methods are not directly comparable, except for the Ladder VAE (LVAE) (Sønderby et al., 2016), since the training is performed differently.”
Towards a Neural Statistician (2016, background): “The approximate posterior over the context given by the statistic network is a function of a sample mean.”
Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “The same problem is reported by (Sønderby et al., 2016) and is a common problem for Bayesian DNNs.”
Structured Inference Networks for Nonlinear State Space Models (2016, positive): “As in (Kaae Sønderby et al. 2016), we found annealing the KL divergence in the variational bound ( L ( (cid:126)x ; ( θ, φ )) ) from 0 to 1 over 5000 parameter updates got better results.”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “A variational free energy approximation/bound on the log-likelihood is then optimized, potentially using the warm-start procedure from (Sønderby et al., 2016) to gradually increase the inﬂuence of regularization eﬀects (at the expense of altering the original bound).”
Batch-shaping for learning conditional channel gated networks (2019, methodology): “Thus, in all our experiments, we introduce this L0-loss after some delay, and we use a warm-up schedule as described in Sønderby et al. (2016)”
VPE: Variational Policy Embedding for Transfer Reinforcement Learning (2018, background): “The KLdivergence term in the loss was linearly increased from zero the first 50 thousand iterations [31].”
Modeling and Transforming Speech Using Variational Autoencoders (2016, background): “The first problem is mentioned in some other works on image and natural language processing [24, 6].”
An Information-Theoretic Analysis of Deep Latent-Variable Models (2017, background): “Snderby et al. (2016) applies the same idea to ladder network decoders.”
Universal audio synthesizer control with normalizing flows (2019, unknown): “VAEs are powerful representation learning frameworks, while remaining simple and fast to learn without requiring large sets of examples [10].”
Avoiding Latent Variable Collapse With Generative Skip Models (2018, methodology): “Many papers describe latent variable collapse (Bowman et al., 2015; Sønderby et al., 2016; Kingma et al., 2016; Chen et al., 2016; Zhao et al., 2017; Yeung et al., 2017).”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, methodology): “, 2017) or equivalently annealing of β (Sønderby et al., 2016), scaling down β (Kingma et al.”
Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models (2018, background): “Note that prior work has loosely suggested that the KL regularizer indigenous to VAEs could potentially mute the impact of superfluous latent dimensions as part of the model optimization process (Burda et al., 2015; Sønderby et al., 2016).”
Improved Conditional VRNNs for Video Prediction (2019, methodology): “Usually this results in models that only use one specific level of the hierarchy [19, 25, 32].”