Inference Suboptimality in Variational Autoencoders

By Chris Cremer, Xuechen Li, David Kristjanson Duvenaud. 2018

312 citations recorded by the source corpus.

Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “Since the marginal log-likelihood only depends on the generative model, overparameterizing the decoder network may hurt the test log-likelihood, whereas powerful encoders can yield better models because of reducing the amortization gap [6].”
Meta-Learning Probabilistic Inference for Prediction (2018, background): “This is due to VI’s tendency to under-fit, especially for small numbers of data points (Trippe and Turner, 2018; Turner and Sahani, 2011) which is compounded when using inference networks (Cremer et al., 2018).”
Bayesian Learning of Neural Network Architectures (2019, methodology): “Note that relaxing the in-dependence and/or the functional form assumption on the network weights can improve modelling performance, as shown by [Cremer et al., 2018, Pawlowski et al., 2017].”
Taming VAEs (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [21, 22, 23, 24, 25, 3, 26, 5] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [27, 28].”
Iterative Amortized Inference (2018, background): “More generally, the inability of inference models to reach fully optimized approximate posterior estimates results in decreased modeling performance, referred to as an amortization gap (Krishnan et al., 2018; Cremer et al., 2017).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, background): “Another thread of research focuses on a different problem called the “amortization gap” (Cremer et al., 2018), which refers to the difference of ELBO caused by parameter sharing of the inference network.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “…since the KL divergence is invariant under the transformation between e←→ z: DKL(q(z|x)||p(z|x)) = DKL(q(e)||p(e|x)) This gives us an alternative interpretation of inference suboptimality (Cremer et al., 2018): the inaccuracy of inferring the true posterior p(z|x) can be attributed to the…”
Likelihood-free MCMC with Approximate Likelihood Ratios (2019, background): “However, this typically leads to an amortization gap [42, 43] which has been extensively studied by [44].”
MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets (2019, methodology): “For perspectives on the balance between variational family and encoder architecture, see Cremer et al. (2018).”
Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, methodology): “The difference between the optimal lower bound L and the optimal (log-)likelihood L of the model without the variational approximation is given by D KL ( p || q (cid:63) ) and is referred to as the approximation gap [77].”
MaxEntropy Pursuit Variational Inference (2019, background): “However, optimizing over a parametric variational family of distributions and getting the optimal solution q∗ = arg max Qλ F [q] still leads to the approximation gap [9], equal to log p(X) − F [q∗].”
Generalized ELBO with Constrained Optimization , GECO (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [6, 7, 8, 9, 10, 11, 12, 13, 14] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [15, 16].”
Deep Mixture Generative Autoencoders (2021, methodology): “There are two measures to evaluate the quality of approximate inference in VAEs [22].”
Advances in Variational Inference (2017, background): “, the error introduced by replacing the local variational parameters by a parametric function [30].”
Faster Attend-Infer-Repeat with Tractable Probabilistic Models (2019, background): “One issue is the nontrivial interplay between the generative model and the inference network, which frequently causes the generative model to adapt to a too weak inference network (Cremer et al., 2018).”
Semi-Amortized Variational Autoencoders (2018, methodology): “Finally, since our work only addresses the amortization gap (the gap between the log-likelihood and the ELBO due to amortization) and not the approximation gap (due to the choice of a particular variational family) (Cremer et al., 2018), it can be combined with existing work on employing richer post”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, unknown): “Additionally, amortized inference leads to an amortization gap, the difference in ELBO for the amortized posterior and optimal approximate posterior [36].”
VoxelMorph: A Learning Framework for Deformable Medical Image Registration (2018, background): “Indeed, the resulting deformation can be interpreted as simply an approximation or initialization to the optimal deformation φ∗, and the resulting difference is sometimes referred to as the amortization gap [15], [66].”