Advances in Variational Inference

By Cheng Zhang, Judith Bütepage, Hedvig Kjellström, Stephan Mandt. 2017

815 citations recorded by the source corpus.

Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem. This approach has been successfully applied to various models and large-scale applications. In this review, we give an overview of recent trends in variational inference. We first introduce standard mean field variational inference, then review recent advances focusing on the following aspects: (a) scalable VI, which includes stochastic approximations, (b) generic VI, which extends the applicability of VI to a large class of otherwise intractable models, such as non-conjugate models, (c) accurate VI, which includes variational models beyond the mean field approximation or with atypical divergences, and (d) amortized VI, which implements the inference over local latent variables with inference networks. Finally, we provide a summary of promising future research directions.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “Aside from direct applications, VAEs embody the success of variational methods in deep learning and have inspired a wide range of ongoing research [23, 46].”
Generalizing from a Few Examples (2019, background): “n image into pixel-wise. The model sequentially generates each pixel conditioned on both already generated pixels and related information acquired from a memory storing Dtrain. (3) Inference networks [141]. Edwards and Storkey [2017] learn one inference networks to infer these latent variables, and ”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “Recent work has experimented with deep probabilistic models to give more reliable uncertainty estimations, including Bayesian neural networks [136, 154], deep neural networks that integrate dropout [47], or ensemble methods to approximate Bayesian inference [83].”
Non-Parametric Calibration for Classification (2019, background): “…joint distribution factorizes as p ( y , g , u ) = p ( y | g ) p ( g | u ) p ( u ) . approximation We aim to ﬁnd a variational approximation q ( u ) = N ( u | m , S ) to the posterior p ( u | y ) . treatments For general on variational inference we refer to Blei et al. (2017), Zhang et al. (2018).”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, methodology): “By parameterizing the posterior and likelihood with our encoder and decoder neural networks, we can amortize our inference costs over the datapoints in our dataset [64].”
GP-VAE: Deep Probabilistic Time Series Imputation (2019, methodology): “Since the exact posterior is intractable, we use variational inference [6, 22, 48].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “The advantage of VAEs is their probabilistic nature, which allows for the assessment of their forecasting ability in terms of a tractable lower bound to the model’s likelihood [55].”
Deep Probabilistic Video Compression (2018, methodology): “We employ amortized variational inference (Blei et al., 2017; Zhang et al., 2017; Marino et al., 2018) to predict a distribution over latent codes given the input video,”
Generalization in Generation: A closer look at Exposure Bias (2019, methodology): “Training via Variational Inference Variational inference (Zhang et al., 2018) allows to optimize a lower-bound instead of the intractable marginal likelihood and has become the standard methodology to training latent variable models.”
Semi-Amortized Variational Autoencoders (2018, background): “…(Ranganath et al., 2014; 2016), and amortized (Mnih & Gregor, 2014; Kingma & Welling, 2014; Rezende et al., 2014) variational inference have made it possible to scale to large datasets and rich, non-conjugate models (see Blei et al. (2017), Zhang et al. (2017) for a review of modern methods).”
On Fenchel Mini-Max Learning (2019, background): “Among them variational inference (VI) [6] and generative flows (FLOW) [56, 53] are two of the most promising directions, and have undergone rapid development recently [66].”
Variational Beam Search for Novelty Detection (2021, methodology): “…exact over s t p β ( s t | t not available as well (by Bayes rule). inference However, we can follow a structured variational approach (Wainwright and Jordan, 2008; Hoffman and Blei, 2015; Zhang et al., 2018), deﬁning a joint variational distribution q ( z t , s t ) = q ( s t ) q ( z t | s t ) .”