On Sparse Variational Methods and the Kullback-Leibler Divergence between Stochastic Processes

By Alexander G. de G. Matthews, James Hensman, Richard E. Turner, Zoubin Ghahramani. 2015

204 citations recorded by the source corpus.

The variational framework for learning inducing variables (Titsias, 2009a) has had a large impact on the Gaussian process literature. The framework may be interpreted as minimizing a rigorously defined Kullback-Leibler divergence between the approximating and posterior processes. To our knowledge this connection has thus far gone unremarked in the literature. In this paper we give a substantial generalization of the literature on this topic. We give a new proof of the result for infinite index sets which allows inducing points that are not data points and likelihoods that depend on all function values. We then discuss augmented index sets and show that, contrary to previous works, marginal consistency of augmentation is not enough to guarantee consistency of variational inference with the original model. We then characterize an extra condition where such a guarantee is obtainable. Finally we show how our framework sheds light on interdomain sparse approximations and sparse approximations for Cox processes.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, background): “Given a dataset D = ( X , y ) , a prior p ( f ) and a variational family Q f , function-space variational inference (Burt et al., 2021; Matthews et al., 2016; Rudner et al., 2021; Sun et al., 2019) consists of finding the variational distribution q ( f ) ∈ Q f that maximizes This variational…”
Streaming Sparse Gaussian Process Approximations (2017, methodology): “In this paper, we focus on the variational free energy approximation scheme [3, 14] which lower bounds the marginal likelihood of the data using a variational distribution q(f) over the latent function:”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, background): “P evaluated at the inputs x˜ t. The KL term for the latent variables H is analytically tractable since both terms are Gaussian. The KL term between the GPs has been shown to simplify to KL q(U)||p(U) [22], which again is analytically tractable since both terms are Gaussian. The expected log-likeliho”
GPflow: A Gaussian Process Library using TensorFlow (2016, positive): “in a Kullback-Leibler sense to the posterior (Matthews et al., 2016).”
Continual Multi-task Gaussian Processes (2019, background): “(7) coincides with the approximated posterior process that VI on sparse GP models aims to minimize through the KL divergence (Matthews et al., 2016).”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, background): “In common with many state-of-the-art GP approximation schemes we start from a sparse inducing point variational framework (Matthews et al., 2016) to achieve computational tractability within each layer, but we do not force independence between the layers.”
Learning Stationary Time Series using Gaussian Processes with Nonparametric Kernels (2015, background): “This formulation can be made technically rigorous for latent functions [20], but we do not elaborate on that here to simplify the exposition.”
Understanding Variational Inference in Function-Space (2020, background): “a KL divergence) is minimized between approximate and posterior predictive distributions (Matthews et al., 2016).”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, methodology): “…inference scheme that makes use of innovations from inducing point GP approximation literature [Titsias, 2009, Hensman et al., 2013, Quiñonero-Candela and Rasmussen, 2005, Matthews et al., 2016, Bui et al., 2017] and previous work on inferring meta-representations [Karaletsos et al., 2018].”
A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (2016, methodology): “The VFE approach can also be extended to employ inter-domain variables (Álvarez et al., 2010; Tobar et al., 2015; Matthews et al., 2016).”
Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes (2020, background): “Variational approaches (Titsias, 2009; Matthews et al., 2016) maximize a lower bound on the log marginal likelihood.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “The interdomain idea has also been applied to the posterior approximations [71], [76], [172].”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “In particular, building on recent work on pseudo-point sparse approximations [Titsias, 2009, Hensman et al., 2015, Matthews et al., 2016, Bui et al., 2017b] and streaming approximations [Csató and Opper, 2002, Bui et al., 2017a], we develop a streaming variational approximation that approximates…”
Understanding Probabilistic Sparse Gaussian Process Approximations (2016, methodology): “[15] show that this procedure approximates the posterior over the entire process f correctly.”
Functional Variational Bayesian Neural Networks (2019, background): “As pointed out in Matthews et al. (2016), it does not have a convenient form as ∫ log q(f)p(f)q(f)df due to there is no infinitedimensional Lebesgue measure (Eldredge, 2016).”
Rethinking Function-Space Variational Inference in Bayesian Neural Networks (2020, background): “Unfortunately, in the absence of additional assumptions about ˜ q ( ˜ f ) and ˜ p ( ˜ f ), this KL divergence is still intractable (de G. Matthews et al., 2016).”