Fast Variational Inference in the Conjugate Exponential Family

By James Hensman, Magnus Rattray, Neil D. Lawrence. 2012

138 citations recorded by the source corpus.

We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic inference using our bound.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “It has been developed for the same class of models described here (Sung et al., 2008; Hensman et al., 2012).”
Gaussian Processes for Big Data (2013, methodology): “In this paper we show how recent advances in variational inference [Hensman et al., 2012, Hoﬀman et al., 2012] can be combined with the idea of inducing variables to develop a practical algorithm for ﬁtting GPs using stochastic variational inference (SVI).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, background): “This is stated below where FIM is denoted by F(η), F(η)−1∇ηL(η) = ∇mL∗(m), (26) This relationship has been discussed in the earlier work of Hensman et al. (2012) and can also be found in Amari (2016).”
Continual Multi-task Gaussian Processes (2019, background): “(2) would require to evaluate expectations of the form E q ( f ) [log p ( y old , y new | f )] , where q ( f ) = p ( f | u ) q ( u | φ ) d u as in the uncollapsed version of the bound (L´azaro-Gredilla and Titsias, 2011; Hensman et al., 2012).”
Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models (2018, background): “[7] in the context of conjugate models, although they do not explicitly mention the connection to duality.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, unknown): “tion [37]. Another interesting property is that taking a unit step in the natural gradient direction equals to performing an update in the Variational Bayes Expectation Maximization (VB-EM) framework [87]. Though showing 6The superiority of natural gradients is veriﬁed in [86]. 7 high scalability an”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “The natural gradients could also be used in other optimization schemes [Hensman et al., 2012, Salimbeni et al., 2018].”
Advances in Variational Inference (2017, methodology): “Recent advances in this direction include [64], [106], [121].”
Probabilistic Line Searches for Stochastic Optimization (2015, methodology): “This includes the training of neural network architectures, online versions of basic algorithms like logistic regression (Zhang, 2004; Bottou, 2010) and more advanced variational models (e.g. Hoffman et al., 2013; Hensman et al., 2012; Broderick et al., 2013).”
Gaussian Processes for Big Data through Stochastic Variational Inference (2012, background): “It can also be shown that taking this unit step is the same as performing a VB update [5, 6].”
Fully simplified multivariate normal updates in non-conjugate variational message passing (2014, unknown): “This relationship is pointed out in Section 4.1 of Hensman et al. (2012).”
Parametric Gaussian process regression for big data (2017, background): “, [17, 18]) can be combined with the idea of inducing variables to develop a practical algorithm for fitting Gaussian processes using stochastic variational inference.”
Integrated Non-Factorized Variational Inference (2013, background): “An active area of research has been focused on developing more efficient and accurate variational inference algorithms, for example, collapsed inference [8, 9], non-conjugate models [10, 11], multimodal posteriors [12], and fast convergent methods [13, 14].”