Variational Autoencoder with Implicit Optimal Priors

By Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, Satoshi Yagi. 2018

70 citations recorded by the source corpus.

The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the standard Gaussian distribution is usually used for the prior, this simple prior incurs over-regularization. As a sophisticated prior, the aggregated posterior has been introduced, which is the expectation of the posterior over the data distribution. This prior is optimal for the VAE in terms of maximizing the training objective function. However, KL divergence with the aggregated posterior cannot be calculated in a closed form, which prevents us from using this optimal prior. With the proposed method, we introduce the density ratio trick to estimate this KL divergence without modeling the aggregated posterior explicitly. Since the density ratio trick does not work well in high dimensions, we rewrite this KL divergence that contains the high-dimensional density ratio into the sum of the analytically calculable term and the lowdimensional density ratio term, to which the density ratio trick is applied. Experiments on various datasets show that the VAE with this implicit optimal prior achieves high density estimation performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Furthermore, classifiers [54, 55, 56], adversarial methods [57], and other techniques [58, 59] have been used to define prior distributions implicitly.”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, background): “Standard Autoencoder Form (RNF)1 Much recent work has demonstrated the sensitivity of VAE performance to the choice of prior distribution, with suboptimal priors either having an “over-regularising” effect on the loss function during training [40]–[42], or leading to posterior collapse [43].”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Takahashi et al. (2019) use the likelihood ratio estimator to train a simple prior distribution.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[73] Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi.”
Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition (2019, methodology): “Even though using the aggregate posterior as the prior is the objective in multiple recent works, it can be challenging in high dimensions, lead to over-fitting and often comes at the expense of additional hyper-parameters [47], [48], [49].”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, unknown): “01 for all models in order to avoid the over-regularization issue [51].”