Learning Hierarchical Priors in VAEs

By Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, Patrick van der Smagt. 2019

110 citations recorded by the source corpus.

We propose to learn a hierarchical prior in the context of variational autoencoders to avoid the over-regularisation resulting from a standard normal prior distribution. To incentivise an informative latent representation of the data, we formulate the learning problem as a constrained optimisation problem by extending the Taming VAEs framework to two-level hierarchical models. We introduce a graph-based interpolation method, which shows that the topology of the learned latent representation corresponds to the topology of the data manifold---and present several examples, where desired properties of latent representation such as smoothness and simple explanatory factors are learned by the prior.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “In deep hierarchical VAEs [5, 9, 4, 42, 43], to increase the expressiveness of both the approximate posterior and prior, the latent variables are partitioned into disjoint groups, zzz = { zzz 1 ,zzz 1 , . . . ,zzz L } , where L is the number of groups.”
LaDDer: Latent Data Distribution Modelling with a Generative Prior (2020, unknown): “A closely related work is [24], where a generative model is used to parameterise the prior.”
Efficient-VDVAE: Less is more (2022, methodology): “To help alleviate this problem, deep HVAEs [19, 20, 21, 22] have been proposed in recent years as a way to increase the expressiveness of both the approximate posterior and the prior distributions, such as:”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “The prior hole problem is commonly tackled by increasing the ﬂexibility of the prior via hierarchical priors (Klushyn et al., 2019), autoregressive models (Gulrajani et al., 2016), a mixture of approximate posteriors (Tomczak & Welling, 2018), normalizing ﬂows (Xu et al., 2019), resampled priors…”
Deep Mixture Generative Autoencoders (2021, methodology): “Another way is by introducing auxiliary variables, such as the hierarchical variational models [14], [28], the Hamiltonian variational inference [29], or using two stochastic layers [30].”