Tighter Variational Bounds are Not Necessarily Better

By Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank D. Wood, Yee Whye Teh. 2018

217 citations recorded by the source corpus.

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Moreover, PIWAE can simultaneously deliver improvements in both the quality of the inference network and generative network, relative to IWAE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, background): “A similar phenomenon was observed in the context of variational autoencoders (VAEs) (Kingma and Welling, 2014), where one maximizes a lower bound on the data likelihood: Looser bounds often yield better inference models, i.e. latent representations (Rainforth et al., 2018).”
Lifelong Generative Modeling (2017, background): “This has been attributed to the fact that VAEs don’t learn the true posterior and make a simplistic assumption regarding the reconstruction distribution Pθ(x|z) [53, 54].”
Importance Weighting and Variational Inference (2018, background): “Rainforth et al. [18, Theorem 1 in Appendix] provide a related asymptotic for errors in gradient variance, assuming at least the third moment exists.”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, methodology): “…such as the importance weighted lower bound used in an IWAE (Burda et al., 2015), or combinations of the VAE and IWAE lower bounds, as proposed in (Rainforth et al., 2018); (iii) alternative approximate inference techniques for estimating the posterior distribution over model parameters; (iv)…”
Semi-Amortized Variational Autoencoders (2018, result): “This is in line with recent work which suggests that training with tighter ELBO does not necessarily result in better inference networks (Rainforth et al., 2018).”
Auto-Encoding Sequential Monte Carlo (2017, background): “In simultaneously developed work (Rainforth et al., 2017), we formalized this intuition in the IWAE setting by showing that the estimator of ∇ φ ELBO IS ( θ, φ, x ) with K particles, denoted by I K , has the following signal-to-noise ratio ( SNR ): We thus see that increasing K reduces the SNR and…”
The Thermodynamic Variational Objective (2019, methodology): “The ELBO (left) is a single partition approximation of the same integral. weighted autoencoder ( IWAE ) objective (Burda et al., 2016) for which tightening of the bound is known to make inference network learning worse (Rainforth et al., 2018).”
MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets (2019, methodology): “Rainforth et al. (2018) give a formal treatment of this ambivalence, and propose modifications of the IWAE objective that tackle it.”
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives (2018, methodology): “However, Rainforth et al. (2018) point out issues with gradient estimators of multi-sample lower bounds.”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “The standard gradient estimator of IWAE can have undesirably high variance (Rainforth et al., 2018; Roeder et al., 2017).”
Semi-Implicit Variational Inference (2018, background): “One may further consult Rainforth et al. (2018) to help select K , J , and ˜ K for SIVI.”
Faster Attend-Infer-Repeat with Tractable Probabilistic Models (2019, background): “On the other hand, and somewhat paradoxically, trying to improve the quality of the variational approximation might also be detrimental for learning (Rainforth et al., 2018).”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “However, IW-VAE has been proven harmful to representation learning, although it provably tightens the likelihood bound [58].”
On Fenchel Mini-Max Learning (2019, background): “While tighter bounds can be achieved for VI via importance re-weighting [10], flexible posteriors [47] and alternative evidence scores [62], these strategies do not necessarily improve performance [55].”
On Nesting Monte Carlo Estimators (2017, result): “As we show in Rainforth et al. (2018), these results further carry over to the reparameterized derivative estimates ∇ θ I N,M .”
Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation (2019, background): “Recent work has investigated tighter objectives [6, 20, 18, 17, 23, 25], based on the following principle: Let R be an estimator of the likelihood—i.”
Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference (2018, methodology): “Independently of our work Rainforth et al. (2017a) has analysed nested Monte Carlo objectives, including the IWAE bound as special case.”
Importance Weighted Hierarchical Variational Inference (2019, methodology): “We also compute SNR on a toy task from (Rainforth et al., 2018) for an upper bound (theorem A.”