Hyperspherical Variational Auto-Encoders

By Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, J. Tomczak. 2018

473 citations recorded by the source corpus.

The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure. To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space. Through a series of experiments we show how such a hyperspherical VAE, or $\mathcal{S}$-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, $\mathcal{N}$-VAE, in low dimensions on other data types.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “Related work on using the von-Mises Fisher distribution for VAE shows that sentences can also be represented using latent vectors on the unit sphere [9, 14, 38].”
Diagnosing and Enhancing VAE Models (2019, neutral): “For one interesting representative example, see (Davidson et al., 2018).”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “The sampling of the vMF distribution follows the procedure of Ulrich (1984); Davidson et al. (2018) and the detailed algorithm is given in Appendix A. 4.3 Model-based MOCA x old f y h g x old f yh g+ fΔSampling x old f yh g+ fΔ xnew Dropout(h) x old f yh g+ fΔ x old xnew or h(x )-h(x)new variation…”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, background): “This lower bound objective is analogous to the recently proposed hyperspherical variational autoencoder (Davidson et al., 2018; Xu and Durrett, 2018).”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, methodology): “Existing works have motivated this choice from a manifold mapping perspective (Liu et al., 2017; Davidson et al., 2018) and computation stability (Xu & Durrett, 2018; Wang et al., 2017).”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “For example, non-parametric Bayesian distribution (Nalisnick and Smyth 2017) and hy-perspherical distribution (Davidson et al. 2018) are used for the prior.”
De Novo Protein Design for Novel Folds using Guided Conditional Wasserstein Generative Adversarial Networks (gcWGAN) (2019, background): “Notably, gcWGAN designs are much more diverse, which can be attributed to that WGAN with its implicit model and WassersteindistanceovercomesknownlimitationsofVAE(YinandZhou, 2018; Arjovsky et al. , 2017; Davidson et al. , 2018).”
Learning with Hyperspherical Uniformity (2021, background): “[51, 52, 49, 48, 50, 80, 79, 20, 19, 60, 14, 47] propose to learn representations on hypersphere and show that angular information in neural networks, in contrast to magnitude, preserves the key semantics and is very crucial to generalization.”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, methodology): “We follow Davidson et al. (2018) and use an acceptance-rejection scheme to sample from the vMF distribution.”
SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations (2020, methodology): “Kingma et al. (2014) and Davidson et al. (2018) propose the stacked VAE structure (M1+M2), which forces the model to utilize the representations learned from ELBO L to inference the q φ ( y | X ) .”
Deep generative models for T cell receptor protein sequences (2019, background): “However, VAE variants do use other distributions in place of normal (Dilokthanakul et al., 2016; Davidson et al., 2018).”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “Despite the remarkable achievements, the behavior of VAEs is still far from satisfactory due to some theoretical and practical challenges such as balancing issue [30] and variable collapse phenomenon [31].”
Spherical Latent Spaces for Stable Variational Autoencoders (2018, methodology): “, 2005), VAE for other domains (Davidson et al., 2018; Hasnat et al., 2017), and a generative editing model for text (Guu et al.”
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning (2019, methodology): “matrix = diag( ˙) (the ‘mean-ﬁeld’ approximation) e.g., [13, 3, 28, 23]. We call this the ‘Cartesian’ posterior. The use of a Cartesian posterior has been motivated by the limit theorem style results [37, 25, 1]. In the common case of a unit multivariate Gaussian prior the ELBO objective is a sum ov”
Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction (2021, methodology): “For link prediction, the prevalent frameworks [32, 48, 12, 59] adopt an auto-encoder formulation, which uses GNNs to encode node representations, and decodes edges as a function over node pairs.”
Explorations in Homeomorphic Variational Auto-Encoding (2018, methodology): “In (Davidson et al., 2018), the reparameterization technique of Naesseth et al.”
SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, background): “Beyond face recognition, the idea of learning a representation on the hypersphere is also shown generally useful in a diverse set of applications, such as few-shot recognition [47], [48], [49], [50], [51], deep metric learning [25], [52], self-supervised learning [53], [54], [55], [56], generative m”
MolGAN: An implicit generative model for small molecular graphs (2018, background): “Several works explored auto-encoder architectures utilizing graph convolutions for link prediction within graphs (Kipf & Welling, 2016b; Grover et al., 2017; Davidson et al., 2018).”