From Variational to Deterministic Autoencoders

By Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, Bernhard Schölkopf. 2019

300 citations recorded by the source corpus.

Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of VAEs. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without forcing it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data, we introduce an ex-post density estimation step that can be readily applied also to existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules. \footnote{An implementation is available at: \url{this https URL}}

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Previous work in this direction learns the latent structure of pre-trained VAEs (Dai & Wipf, 2019; Xiao et al., 2019; Ghosh et al., 2020), and sampling from learned latent distributions improves sample quality.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “One surprising aspect is that VAEs have been challenged on both of its own design functionalities, as generative models [14, 12] and as log-likelihood optimizers [30, 33].”
Perceptual Generative Autoencoders (2019, methodology): “The first block shows the results from Ghosh et al. (2019), where CV-VAE stands for constant-variance VAE, and RAE stands for regularized autoencoder.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, background): “In contrast, GANs do not suffer from this problem [17].”
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images (2019, methodology): “The RAE consists of a convolutional and deconvolutional trunk of 4 layers of 32 ﬁlters each, with 3 × 3 kernel size.”
Image-to-Image Translation: Methods and Applications (2021, background): “As stated in [74], VAEs provide more stable training than”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, unknown): “Related to VAEs are regularised autoencoders (RAEs) which apply regularisation to the latent space of a deterministic autoencoder then subsequently train a density estimator on this space to obtain a complex prior [57].”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “, 2017), or training a posthoc prior (Engel et al., 2017; Ghosh et al., 2019).”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “3] learns an implicit prior distribution in the latent space with adversarial training and demonstrates superior performance on text generation. Recently, some papers resort to a two-stage approach 3 [12, 16]. They ﬁrst train a VAE or deterministic autoencoder (with some form of regularization) in t”
On the Sentence Embeddings from BERT for Semantic Textual Similarity (2020, background): “This is a common problem in the context of representation learining (Rezende and Viola, 2018; Li et al., 2019; Ghosh et al., 2020).”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “[1, 103, 104] introduce regularisation to the latent space of the encoder and lead to a smooth and representative latent space without conforming to an arbitrarily chosen prior.”
Learning to Dress 3D People in Generative Clothing (2019, unknown): “The sampling trick in [14] is used.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Regularized autoencoders (RAE) (Ghosh et al., 2020) train an autoencoder, and subsequently a Gaussian mixture model on latent codes.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “They cast learning representations for high-dimensional distributions as a VI problem [123].”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “tent space relatively smooth. To prevent degenerated mapping where latent space is free of any structure, there is a rich literature on restricting the low-energy area for data points on the manifold [34, 35, 36, 37, 38]. Here we adopt the popular variational autoencoder framework [36, 27] to optimi”
On the Evaluation of Conditional GANs (2019, background): “Conditioning Modality Embedding Function Class / Attribute labels One-hot encoding Bounding boxes / Masks Regularized AE (Ghosh et al., 2019) Images Inceptionv3 (Szegedy et al.”