Mean field variational Bayes for continuous sparse signal shrinkage: Pitfalls and remedies

By S. Neville, John Thomas Ormerod, Matt P. Wand. 2014

67 citations recorded by the source corpus.

We investigate mean field variational approximate Bayesian in- ference for models that use continuous distributions, Horseshoe, Negative- Exponential-Gamma and Generalized Double Pareto, for sparse signal shrin- kage. Our principal finding is that the most natural, and simplest, mean field variational Bayes algorithm can perform quite poorly due to poste- rior dependence among auxiliary variables. More sophisticated algorithms, based on special functions, are shown to be superior. Continued fraction ap- proximations via Lentz's Algorithm are developed to make the algorithms

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “…have been developed for a variety of classes of models, including shrinkage models (Armagan, Clyde, and Dunson 2011; Armagan and Dunson 2011; Neville, Ormerod, and Wand 2014), general time-series models (Roberts et al. 2004; Barber and Chiappa 2006; Archambeau et al. 2007a, 2007b; Johnson…”
Bayesian Compression for Deep Learning (2017, methodology): “Instead of directly working with the half-Cauchy priors we will employ a decomposition of the half-Cauchy that relies upon (inverse) gamma distributions [Neville et al., 2014] as this will allow us to compute the KL-divergence between the scale prior p(z) and a log-normal scale posterior qφ(z) in cl”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “However, auxiliary variables introduced by the decomposition can pose further inferential challenges and lead to poorer approximations (Neville et al., 2014).”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Some examples, with explanations for the inaccuracy, are given in Wang and Titterington (2005) and Neville et al. (2014).”
A variational Bayes approach to variable selection (2017, background): “The VB approach is summarized in many places including Bishop (2006); Rue et al. (2009); Ormerod and Wand (2010); Carbonetto and Stephens (2011); Faes et al. (2011); Wand et al. (2011); Murphy (2012); Pham et al. (2013); Luts and Ormerod (2014); Neville et al. (2014); You et al. (2014).”