Gaussian variational approximation with sparse precision matrices

By Linda S. L. Tan, David J. Nott. 2016

89 citations recorded by the source corpus.

We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods that make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients, which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state-space models for time series.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, background): “Gaussian variational distributions with full covariance matrices have been used extensively for shallow models [6, 8, 16, 22, 24, 30, 34, 35].”
Variance reduction properties of the reparameterization trick (2018, methodology): “Some papers in the literature explain the success of the RP trick as due to its efficient use of gradient information from the log-joint density (Titsias and LázaroGredilla, 2014; Tan and Nott, 2018; Quiroz et al., 2018) without elaborating further.”
Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference (2017, methodology): “In particular, Tan and Nott [2017] 2: Results on variational (VAE) and importance-weighted (IWAE) autoencoders using the total derivative estimator, equation (7), versus the path derivative estimator, equation (8) (ours). eliminating certain terms from a gradient estimator for Gaussian families…”