Symmetric Variational Autoencoder and Connections to Adversarial Learning

By Yunchen Pu, Liqun Chen, Shuyang Dai, Weiyao Wang, Chunyuan Li, Lawrence Carin. 2017

73 citations recorded by the source corpus.

A new form of the variational autoencoder (VAE) is proposed, based on the symmetric Kullback-Leibler divergence. It is demonstrated that learning of the resulting symmetric VAE (sVAE) has close connections to previously developed adversarial-learning methods. This relationship helps unify the previously distinct techniques of VAE and adversarially learning, and provides insights that allow us to ameliorate shortcomings with some previously developed adversarial methods. In addition to an analysis that motivates and explains the sVAE, an extensive set of experiments validate the utility of the approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems (2019, methodology): “Finally, for the discriminator we use 4 convolution layers with the channel sizes of [ 32 , 64 , 128 , 256 ] , each with kernel size of 5, stride 2, and a hyperbolic tangent activation function in all layers except the last.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “Our algorithm is different from existing hybrid models which train generator and inference models using a single optimization function [36] or learn an optimal coupling between generator and inference model by using adversarial learning [11, 15, 16, 36, 37, 42, 43, 61].”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “The recently proposed method [29] shares the same spirit.”
Adversarial Uncertainty Quantification in Physics-Informed Neural Networks (2018, background): “…theory, the optimal set of parameters { θ ∗ , φ ∗ , ψ ∗ } correspond to the Nash equilibrium of the two player game deﬁned by the loss functions in equations 10,11, for which one can show that the exact model distribution and the exact posterior over the latent variables can be recovered [30, 38].”
Inclusive GAN: Improving Data and Minority Coverage in Generative Models (2020, background): “ALI [12], BiGAN [11], and SVAE [8] propose to instead jointly match the bidirectional mappings between data and latent distributions.”
Deep Mixture Generative Autoencoders (2021, background): “Mixed GAN and VAE architectures have been enabled with representation learning capabilities [16], [17].”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, unknown): “ELBO is an important indicator of the VAE’s performance and is used as its main optimization function [11].”
On Fenchel Mini-Max Learning (2019, background): “However, with such learning, samples drawn x ∼ pθ̂(x|z) with z ∼ p(z) may not be as close to the training data as desired [12].”
Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow (2018, methodology): “Makhzani et al. (2016) used adversarial discriminators during the training of VAEs to encourage the marginal distribution of the latent encoding to be similar to the prior distribution, similar techniques include Mescheder et al. (2017) and Chen et al. (2018).”
Learning Disentangled Representations with Reference-Based Variational Autoencoders (2019, unknown): “To address this limitation, we propose to optimize an alternative variational expression inspired by unsupervised Symmetric VAEs (Pu et al., 2018).”
Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, background): “The Variational Autoencoder (VAE) (Kingma and Welling 2013) is a popular generative deep learning model with remarkable successes in learning unsupervised tasks by inferring probabilistic data representations (Chen et al. 2018), for disentangled representation learning (Higgins et al.”
Towards Better Representations with Deep/Bayesian Learning (2018, background): “…VAEs and GANs (Pu et al., 2015; Stevens et al., 2017; Fan et al., 2016a; Pu et al., 2016a,b; Chen et al., 2018b; Li et al., 2018a; Pu et al., 2017b; Chen et al., 2018a; Xu et al., 2017; Wang et al., 2017b; Pu et al., 2018b; Yuan et al., 2017, 2018; Chapfuwa et al., 2018), sigmoid belief networks…”
JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets (2018, background): “Repre-sentative work includes the AAE (Makhzani et al., 2015), VAE-GAN (Larsen et al., 2015), AVB (Mescheder et al., 2017), AS-VAE (Pu et al., 2017), SVAE (Chen et al., 2018), etc .”
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model (2020, methodology): “ines the learning of the generator model and inference model in an adversarial framework. It can be improved by adding conditional entropy regularization as in more recent methods ALICE [26] and SVAE [4]. Though these methods are trained using joint discriminator on image and latent vector, such a d”