Natural Conjugate Gradient in Variational Inference

By Antti Honkela, Matti Tornio, Tapani Raiko, Juha Karhunen. 2007

51 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, methodology): “In Honkela et al. (2008, 2010) a similar idea is proposed in the context of variational inference and speciﬁc assumptions on the form of p ( θ ) are made.”
Stochastic variational inference (2012, methodology): “Researchers have used the natural gradient in variational inference for nonlinear state space models (Honkela et al., 2008) and Bayesian mixtures (Sato, 2001).”
Variational Inference: A Review for Statisticians (2016, background): “For more on natural gradients and variational inference, see Sato (2001) and Honkela et al. (2008).”
Automated Variational Inference in Probabilistic Programming (2013, background): “Natural conjugate gradients for variational inference are investigated in [25], but the analysis is mostly devoted to the case where the variational approximation is Gaussian, and the resulting gradient equation involves an integral which is not necessarily tractable.”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Gaussian Q families have also been used by Lappalainen and Honkela (2000), Archambeau et al. (2007), Raiko et al. (2007), Nickisch and Rasmussen (2008), Honkela and Valpola (2005), Honkela et al. (2007), Honkela et al. (2008) and Opper and Archambeau (2009).”
Natural Gradient Deep Q-learning (2018, methodology): “Insights into the mathematics of optimization using natural conjugate gradient techniques are provided in the work of Honkela et al. [2015].”
Advances in Variational Inference (2017, background): “Natural gradients, first studied in [5] and introduced to VI in [69], [70], [171], take the information geometry of the”
Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes (2010, methodology): “…(Barber and Bishop, 1998), kernel classifiers (Seeger, 2000), nonlinear factor analysis (Lappalainen and Honkela, 2000; Honkela and Valpola, 2005; Honkela et al., 2007) and nonlinear dynamical models (Valpola and Karhunen, 2002; Archambeau et al., 2008), non-conjugate variance models (Valpola et…”
Riemann manifold Langevin and Hamiltonian Monte Carlo methods (2011, background): “Liu (2001), Neal (1993b, 1996), Gustafson (1997), Ishwaran (1999), Husmeier et al. (1999) and Hanson (2001), indicating that it has not been widely adopted as a practical inference method.”
A Stochastic Variational Framework for Fitting and Diagnosing Generalized Linear Mixed Models (2012, methodology): “Honkela et al. (2008) also showed that replacing the ordinary gradient in the conjugate gradient algorithm with the natural gradient can speed up variational learning.”
Scalable Recommendation with Poisson Factorization (2013, background): “We found that following these natural gradients [15, 16, 30] (with step size one) resulted in a better way to fit the objective.”
Non-conjugate Variational Message Passing for Multinomial and Binary Regression (2011, methodology): “Various authors have used standard numerical optimization techniques [15, 17, 3], or adapted such methods to be more suitable for this problem [7, 8].”