Variational Autoencoders Pursue PCA Directions (by Accident)

By Michal Rolinek, Dominik Zietlow, Georg Martius. 2018

176 citations recorded by the source corpus.

The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Toward Causal Representation Learning (2021, background): “Additional assumptions or biases can help favoring certain solutions over others [158], [205].”
Towards Causal Representation Learning (2021, neutral): “help favoring certain solutions over others [158, 205].”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “[37] also discovered the same phenomenon in β -VAE.”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, methodology): “VAEs attempt to estimate the lower bound on the joint distribution of the data and the latent factors p ( x , z ) by optimising the following objective: where, in the usual case, the prior p ( z ) is chosen to be an isotropic unit Gaussian.”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “[36] observed that the diagonal covariance used in the variational distribution of VAEs encourages orthogonal representations.”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, background): “[35] Michal Rolinek, Dominik Zietlow, and Georg Martius.”
Efficient-VDVAE: Less is more (2022, background): “Fortunately, in practice, VAEs (and HVAEs) train in a polarized regime and learn a latent space where the majority of the latent dimensions are "turned off" for all samples [51, 52].”
Disentangling Disentanglement in Variational Autoencoders (2018, unknown): “As noted by Rolinek et al. (2018); Stühmer et al. (2019), increasing β can reinforce existing inductive biases, wherein mean-ﬁeld assumptions encourage representations which reduce dependence between the latent dimensions (Turner and Sahani, 2011).”