Tackling Over-pruning in Variational Autoencoders

By Serena Yeung, Anitha Kannan, Yann Dauphin, Li Fei-Fei. 2017

64 citations recorded by the source corpus.

Variational autoencoders (VAE) are directed generative models that learn factorial latent variables. As noted by Burda et al. (2015), these models exhibit the problem of factor over-pruning where a significant number of stochastic factors fail to learn anything and become inactive. This can limit their modeling power and their ability to learn diverse and meaningful latent representations. In this paper, we evaluate several methods to address this problem and propose a more effective model-based approach called the epitomic variational autoencoder (eVAE). The so-called epitomes of this model are groups of mutually exclusive latent factors that compete to explain the data. This approach helps prevent inactive units since each group is pressured to explain the data. We compare the approaches with qualitative and quantitative results on MNIST and TFD datasets. Our results show that eVAE makes efficient use of model capacity and generalizes better than VAE.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Recently, to improve upon well-known pitfalls like over-pruning (Yeung et al., 2017) of the VAE, Hoffman (2017) proposed HMC-DLGM.”
Variational Russian Roulette for Deep Bayesian Nonparametrics (2019, background): “, 2017), which is also called over-pruning (Burda et al., 2015; Yeung et al., 2017).”
Learning Hierarchical Priors in VAEs (2019, unknown): “Furthermore, we evaluate whether REWO prevents over-pruning of the latent variables (Yeung et al., 2017).”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “Several other papers have studied alternative formulations of the VAE objective [34, 13, 2, 29, 41].”
Constrained Graph Variational Autoencoders for Molecule Design (2018, methodology): “Note that we allow deviation from the pure VAE loss ( λ 1 = 1 ) following Yeung et al. [34].”
A RAD approach to deep mixture models (2019, methodology): “A standard way to address this issue has been to use mixture models (Yeung et al., 2017; Richardson & Weiss, 2018; Eghbal-zadeh et al., 2018) or structured priors (Johnson et al.”
Distributionally Adversarial Attack (2018, methodology): “, 2018), which uses the KL divergence as the energy functional, we adopt the annealing idea in variational autoencoder (Yeung et al., 2017), and introduces an additional term from the loss to define a new energy functional as:”
Avoiding Latent Variable Collapse With Generative Skip Models (2018, background): “Many papers describe latent variable collapse (Bowman et al., 2015; Sønderby et al., 2016; Kingma et al., 2016; Chen et al., 2016; Zhao et al., 2017; Yeung et al., 2017).”