Spherical Latent Spaces for Stable Variational Autoencoders

By Jiacheng Xu, Greg Durrett. 2018

224 citations recorded by the source corpus.

A hallmark of variational autoencoders (VAEs) for text processing is their combination of powerful encoder-decoder models, such as LSTMs, with simple latent distributions, typically multivariate Gaussians. These models pose a difficult optimization problem: there is an especially bad local optimum where the variational posterior always equals the prior and the model does not use the latent variable at all, a kind of “collapse” which is encouraged by the KL divergence term of the objective. In this work, we experiment with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere. With this choice of prior and posterior, the KL divergence term now only depends on the variance of the vMF distribution, giving us the ability to treat it as a fixed hyperparameter. We show that doing so not only averts the KL collapse, but consistently gives better likelihoods than Gaussians across a range of modeling conditions, including recurrent language modeling and bag-of-words document modeling. An analysis of the properties of our vMF representations shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “Related work on using the von-Mises Fisher distribution for VAE shows that sentences can also be represented using latent vectors on the unit sphere [9, 14, 38].”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “Particularly, the development of variational autoencoders [5] has enabled many applications, ranging from image processing to language modeling [6]–[8].”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, background): “This lower bound objective is analogous to the recently proposed hyperspherical variational autoencoder (Davidson et al., 2018; Xu and Durrett, 2018).”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, methodology): “Existing works have motivated this choice from a manifold mapping perspective (Liu et al., 2017; Davidson et al., 2018) and computation stability (Xu & Durrett, 2018; Wang et al., 2017).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “…et al., 2018), deploying less powerful de-coders (Yang et al., 2017; Semeniuta et al., 2017), using lossy input (Chen et al., 2017), utilizing different latent variable connections (Dieng et al., 2017; 2018; Park et al., 2018), or changing the prior (Tomczak & Welling, 2018; Xu & Durrett, 2018).”
Disentangling Disentanglement in Variational Autoencoders (2018, methodology): “independent factors of variation (Alemi et al., 2017; Ansari and Soh, 2019; Burgess et al., 2018; Chen et al., 2018; 2017; Eastwood and Williams, 2018; Esmaeili et al., 2019; Higgins et al., 2016; Kim and Mnih, 2018; Xu and Durrett, 2018; Zhao et al., 2017), typically evaluating this using purposebu”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, methodology): “In order to disentangle the syntactic and semantic aspects of paraphrase generation we learn an explicit latent variable model using a variational autoencoder (VAE) (Kingma and Welling, 2014), which is now commonly applied to text generation (Bowman et al., 2016; Miao et al., 2016; Semeniuta et al.,”
Latent Normalizing Flows for Discrete Sequences (2019, methodology): “In these models, the KL term accounts for less than 5% of the ELBO (Bowman et al., 2016; Kim et al., 2018; Xu & Durrett, 2018), or less than 30% of the ELBO when using a specially designed auxiliary loss (Goyal et al., 2017).”
Mixture Content Selection for Diverse Sequence Generation (2019, unknown): “Dieng et al., 2018; Xu and Durrett, 2018; He et al., 2019; Razavi et al., 2019).”