Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes

By Ryan Giordano, Tamara Broderick, Michael I. Jordan. 2015

89 citations recorded by the source corpus.

Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, a well known major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We generalize linear response methods from statistical physics to deliver accurate uncertainty estimates for model variables—both for individual variables and coherently across variables. We call our method linear response variational Bayes (LRVB). When the MFVB posterior approximation is in the exponential family, LRVB has a simple, analytic form, even for non-conjugate models. Indeed, we make no assumptions about the form of the true posterior. We demonstrate the accuracy and scalability of our method on a range of models for both simulated and real data.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, neutral): “…et al. (2017), Challis and Barber (2013), and Tan and Nott (2017) took advantage of Gaussian variational families with nondiagonal covariance; Giordano, Broderick, and Jordan (2015) post-processed the mean-field parameters to correct for underestimating the variance; and Ranganath, Tran, and…”
Hierarchical Variational Models (2015, methodology): “Giordano et al. (2015) use linear response theory to recover covariances from mean-ﬁeld estimates.”
Copula variational inference (2015, background): “[5], which is to estimate the posterior covariance for a Gaussian mixture.”
Uncertainty in Deep Learning (2016, background): “Several solutions exist for this (such as [Giordano et al., 2015]), with different limitations for their practicality.”
Semi-Implicit Variational Inference (2018, methodology): “…methods that add some structure back to Q to partially restore the dependencies (Saul & Jordan, 1996; Jaakkola & Jordan, 1998; Hoffman & Blei, 2015; Giordano et al., 2015; Tran et al., 2015; 2016; Han et al., 2016; Ranganath et al., 2016; Maaløe et al., 2016; Gregor et al., 2015), it is still…”
Doubly Semi-Implicit Variational Inference (2018, background): “They can be divided roughly into two main groups: those that require the probability density function of the approximate posterior to be analytically tractable (which we will call explicit models) [30, 12, 9, 5, 36, 38, 6, 27, 20] and those that do not (implicit models) [11, 22, 37, 17, 21, 31, 40].”
Bayesian weighted Mendelian randomization for causal inference based on summary statistics (2018, methodology): “(9) Then, according to LRVB, we can obtain the accurate estimate Σ̂ of the covariance matrix of sufficient statistics in BWMR by Σ̂ = −(H−V−1)−1 = (I−VH)−1V. (10) In summary, from VEM algorithm, we obtain the estimate of β as the mean of latent variable β under variational posterior distribution aft”
JuMP: A Modeling Language for Mathematical Optimization (2015, methodology): “JuMP’s unique mix of functionality has driven its growing adoption by researchers [7, 34, 50, 72, 46], and JuMP has been used for teaching in courses in at least 10 universities (e.”