Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

By James Lucas, George Tucker, Roger Baker Grosse, Mohammad Norouzi. 2019

226 citations recorded by the source corpus.

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jan Kautz
3. Ming-Hsuan Yang
4. Jitendra Malik
5. Dacheng Tao
6. Trevor Darrell
7. Arash Vahdat
8. Xiaogang Wang

In Statistical and Nonlinear Physics

1. Jan Kautz
2. Aaron Hertzmann
3. Arash Vahdat
4. Leonidas J. Guibas
5. Saket Navlakha

In Representation & Self-Supervised Learning

1. Aaron Hertzmann
2. Jan Kautz
3. Tolga Birdal
4. Mete Ozay
5. Arash Vahdat
6. Marco Toldo
7. Leonidas J. Guibas
8. Daniel Cohen-Or

In Probabilistic & Bayesian Methods

1. Jan Kautz
2. Aaron Hertzmann
3. Mete Ozay
4. Arash Vahdat
5. Marco Toldo
6. Leonidas J. Guibas

Citations in the corpus

• NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
• Unsupervised anomaly localization using VAE and beta-VAE (2020, background): “Intuitively, a bigger β , putting less weight on the reconstruction accuracy, may cause insensitive reconstruction change with respect to the input change and then less accurate reconstruction, which is similar to posterior collapse [13].”
• Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “An alternative discussion of this connection in the context of linear VAEs is also presented by Lucas et al. (2019).”
• HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “A common difficulty in training VAEs is posterior collapse [21] – when the learned latent encoding zt is effectively ignored by the decoder.”
• Bring Evanescent Representations to Life in Lifelong Class Incremental Learning (2022, background): “Modeling drifts with GMs and VMs: Our framework enables implementation of drift models using different GMs and VMs.”