InfoVAE: Balancing Learning and Inference in Variational Autoencoders

By Shengjia Zhao, Jiaming Song, Stefano Ermon. 2019

338 citations recorded by the source corpus.

A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (Info-VAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “In so doing, Info-VAE minimizes the maximummean discrepancy (MMD) distance (or the KL divergence) between the marginal of the inference network and the prior to implicitly increase the corresponding mutual information contained in the model.”
Preventing Posterior Collapse with delta-VAEs (2019, background): “Most common approaches either change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017), or weaken the decoder (Bowman et al., 2015; Gulrajani et al., 2016).”
Deep Mixture Generative Autoencoders (2021, methodology): “We implement InfoVAE by using a large network architecture.”
Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data (2020, methodology): “We adopt the InfoVAE [52] objective function, and modify it to use discrete latent states in a conditional formulation (since the model uses a CVAE).”
Learning joint latent representations based on information maximization (2021, result): “The results are reported in Table 2, where all other results are cited from [25], except for Infor-VAE [44].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “latents to capture global structure [28]; a mutual information term can be added to the loss to encourage high correlation between x and z [259]; encouraging the posterior to be diverse by controlling its geometry to evenly covering the data space, redundancy is reduced and latents are encouraged to”