Rényi Divergence Variational Inference

By Yingzhen Li, Richard E. Turner. 2016

304 citations recorded by the source corpus.

This paper introduces the variational R\'enyi bound (VR) that extends traditional variational inference to R\'enyi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is controlled by the value of alpha that parametrises the divergence. The reparameterization trick, Monte Carlo approximation and stochastic optimisation methods are deployed to obtain a tractable and unified framework for optimisation. We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework. Experiments on Bayesian neural networks and variational auto-encoders demonstrate the wide applicability of the VR bound.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, methodology): “Other initialisation settings are largely taken from (Li & Turner, 2016).”
VAE with a VampPrior (2017, methodology): “Evaluation We compared the one-layered VAE with the VampPrior, VAE (L = 1) + VampPrior, and the two-layered VAE with the VampPrior, VAE (L = 2) + VampPrior, to other methods with comparable architectures composed of MLPs, such as, Variational Auto-Encoder with the standard normal prior (VAE) [14, 28”
Improving Variational Auto-Encoders using Householder Flow (2016, background): “, importance weighted VAE [4], Rényi Divergence for VAE [12] and Ladder VAE [18].”
Automated Scalable Bayesian Inference via Hilbert Coresets (2017, methodology): “Note that since the coresets approach is agnostic to the particular inference algorithm used, its beneﬁts apply to the continuing developments in both MCMC (Robert et al., 2018) and variational (Dieng et al., 2017; Li and Turner, 2016; Liu and Wang, 2016) approaches.”
Deep and Hierarchical Implicit Models (2017, background): “There are many choices of variational objectives that measure closeness (Ranganath et al., 2016a; Li & Turner, 2016; Dieng et al., 2016).”
Variational Implicit Processes (2018, background): “…variational inference again for scalable stochastic optimization, and we follow [37,54,57] to approximate (9) by the α -energy (see Appendix B): When α → 0 the α -energy reduces to the variational lower-bound, and empirically the α -energy has better approximation accuracy when α > 0 [37, 54, 57].”
Advances in Variational Inference (2017, methodology): “a-divergences have recently been used in variational inference [103], [104].”
Hierarchical Implicit Models and Likelihood-Free Variational Inference (2017, background): “There are many choices of variational objectives that measure closeness [42, 29, 10].”
AUTO-ENCODING VARIATIONAL BAYES (2020, unknown): “Better performance perhaps could be achieved with a family of divergences that interpolates between forward KL and χ(2), such as the Rènyi divergence family [47].”
A Brief Introduction to Machine Learning for Engineers (2017, methodology): “Another generalization based on the use of Renyi divergence in lieu of the KL divergence is presented in [62].”
On Fenchel Mini-Max Learning (2019, background): “, contrastive divergence (CD) [33]), Renyi [40], importance-weighted ELBO [10], and the proposed FML.”
Approximate Inference for Deep Latent Gaussian Mixtures (2016, background): “Burda et al. [2] proposed an importance weighted objective, and Li and Turner [10] then generalized the importance sampling approach to a family of α-divergences.”
MaxEntropy Pursuit Variational Inference (2019, methodology): “ut additional signicant costs of implementing or reformulating the initial statistical problem. Moreover, we could provide intuition for selecting by establishing a connection with Renyi divergence [16] thanks to the analyses of the form of (3). Namely, we consider a parametric mapping in the probab”
An Introduction to Variational Autoencoders (2019, background): “Other objectives have been proposed such as Rényi divergence variational inference (Li and Turner, 2016), Generative Moment Matching Networks (Li et al.”
Reparameterization Gradient for Non-differentiable Models (2018, background): “One interesting future direction is to investigate the possibility of applying our ideas to recent variational objectives [24, 17, 19, 16, 25], which are based on tighter lower bounds of marginal likelihood than the standard ELBO.”