The Variational Gaussian Approximation Revisited

By Manfred Opper, Cédric Archambeau. 2009

410 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Continual Learning (2017, background): “This equivalence also does not hold in the general case, as discussed by Opper and Archambeau (2009) that the Gaussian variational approximation is like an averaged and smoothed Laplace approximation.”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “…updates for µ k and Λ k : where The solution to this optimization problem is given by: To compute ∇ µ k L and ∇ Λ k L , we make use of the following identities (Opper and Archambeau, 2009): Applying these identities to compute the gradients of L (Equation 81), we ﬁnd Using the fact that d Λ k = −…”
Generalized Variational Continual Learning (2020, methodology): “Next, take derivatives wrt Σ T and set to 0: We move from Equation 6 to Equation 7 using Equation 19 in Opper & Archambeau (2008).”
Weight Uncertainty in Neural Networks (2015, methodology): “We apply Proposition 1 to the optimisation problem in (1): let f ( w , θ ) = log q ( w | θ ) − log P ( w ) P ( D| w ) .”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “Given the prior distribution p(w) := N (w|0, δIP ), a minimum of this objective, denoted by μ∗ and Σ∗, satisfies the following condition as shown by [18]:”
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models (2014, background): “Opper & Archambeau (2009) make use of an unconditional variational approximation, which typically requires several iterations for every visible pattern v in order to converge to a local minimum of the free energy.”
Automatic Differentiation Variational Inference (2016, methodology): “However, using a Gaussian variational approximation is not equivalent to the Laplace approximation (Opper and Archambeau, 2009).”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, methodology): “…the variational Bayes optimization problems have been used in the cases of linear models (Knowles & Minka, 2011; Cseke, Opper, & Sanguinetti, 2013) and gaussian processes (Opper & Archambeau, 2009). t However, as we will show here, using nondiagonal covariance can significantly improve performance.”
Scalable Variational Gaussian Process Classification (2014, methodology): “The original model (1) is recovered by marginalization of the additional latent vector: We can now proceed by using a variational sparse GP bound on g , followed by a further variational approximation to deal with the non-Gaussian likelihood.”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, positive): “To derive VON, we use the Bonnet’s and Price’s theorems (Opper & Archambeau, 2009; Rezende et al., 2014) to express the gradients of the expectation of f(θ) with respect to µ and Σ in terms of the gradient and Hessian of f(θ), i.e., ∇µEq [f(θ)] = Eq [∇θf(θ)] := Eq [g(θ)] , (45) ∇ΣEq [f(θ)] = 12Eq […”
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression (2012, background): “For Gaussian q ( x ) and twice diﬀerentiable log p ( x, y ), Minka (2001) and Opper and Archambeau (2009) show that and where ∇ x ∇ x log p ( x, y ) denotes the Hessian matrix of log p ( x, y ) in x .”
Noisy Natural Gradient as Variational Inference (2017, background): “However, in the case of Gaussian distribution with parameters φ = { µ , Σ } , there is another estimator given by Opper & Archambeau (2009): which are due to Bonnet (1964) and Price (1958), respectively.”
GPflow: A Gaussian Process Library using TensorFlow (2016, methodology): “To this end we support ‘variationally sparse’ methods which ensure that the approximation is scalable and close (Opper and Archambeau, 2009), SGPR (Titsias, 2009), SVGP (Hensman et al., 2013, 2015b) and SGPMC (Hensman et al., 2015a).”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, methodology): “We used GPflow [22] with variational Gaussian approximation [24] and L-BFGS-B optimization [41] for GP inference, and utilized mean predictions as pseudo supervisions.”
State Space Gaussian Processes with Non-Gaussian Likelihood (2018, methodology): “…approximation (LA, Williams & Barber, 1998), variational Bayes (VB, Gibbs & MacKay, 2000), direct Kullback–Leibler (KL) divergence minimization (Opper & Archambeau, 2009) and expectation propagation (EP, Minka, 2001) with different tradeoffs in terms of accuracy and required computations (Kuss…”
Variational Heteroscedastic Gaussian Process Regression (2011, methodology): “In this work, we introduce a novel variational inference method for HGP regression that is based on variational Bayes and the Gaussian approximation (Opper & Archambeau, 2009; Nickisch & Rasmussen, 2008; Honkela et al., 2011).”
Integrated Non-Factorized Variational Inference (2013, methodology): “In case it is not available, we may further constrain q(x|y,θk) to a parametric form, examples including: (i) multivariate Gaussian [17], if the posterior asymptotic normality holds; (ii) skew-normal densities [6, 18]; or (iii) an inducing factorization assumption (see Ch.”
Gaussian variational approximation with sparse precision matrices (2016, background): “The relationship between the Laplace and the Gaussian variational approximation is discussed in Opper and Archambeau (2009) while Challis and Barber (2013) consider some different parameterizations in terms of the Cholesky.”