Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes

By Antti Honkela, Tapani Raiko, Mikael Kuusela, Matti Tornio, Juha Karhunen. 2010

114 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “Such large steps may be present with SGD, possibly yielding the model to overﬁt (e.g. getting forced into some quadrant of parameter space based only on a few examples) resulting in different models at the end.”
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data (2016, background): “Ghahramani & Hinton (1996); Honkela et al. (2010) based on the expectation maximization (EM) algorithm or Valpola & Karhunen (2002), which uses neural networks.”
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression (2012, methodology): “Algorithm 1 Stochastic Optimization for Fixed-Form Variational Bayes Require: An unnormalized posterior distribution p ( x, y ) Require: A type of approximating posterior q η ( x ) Require: The total number of iterations N Initialize ˜ η 1 to a ﬁrst guess, for example by matching the prior p ( x ) o”
Advances in Variational Inference (2017, background): “Natural gradients, ﬁrst studied in [5] and introduced to VI in [69], [70], [171], take the information geometry of the model into account.”
Non-conjugate Variational Message Passing for Multinomial and Binary Regression (2011, methodology): “Various authors have used standard numerical optimization techniques [15, 17, 3], or adapted such methods to be more suitable for this problem [7, 8].”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Some recent articles on this topic have used the terms fixed-form variational Bayes (Honkela et al., 2010) and non-conjugate variational message passing (Knowles & Minka, 2011), to describe this modification of mean field variational Bayes.”
Iterative Refinement of Approximate Posterior for Training Directed Belief Networks (2015, background): “Optimization then is dependent on the choice of a parametric form for the posterior that can best approximate the exact posterior, and models generally vary in choice of parameterization followed by a learning algorithm which efficiently maximizes its lower bound.”
Kullback-Leibler Proximal Variational Inference (2015, methodology): “Relationship to the methods that use natural gradients: An alternative approach to incorporate the geometry of the posterior distribution is to use natural gradients [6, 5, 1].”
Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions (2015, methodology): “The Riemannian conjugate-gradient method of Honkela et al. (2011) has similar issues, in that computing ∇ 2 A ( λ ) is typically very costly.”
Iterative Refinement of Approximate Posterior for Training Directed Belief Networks (2015, background): “Coordinate ascent mean ﬁeld inference is expensive, though in limited settings the natural gradient can be used (Honkela et al., 2010; Hoffman et al., 2013).”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “Abbreviations of the inference scheme are: Automatic Differentiation VI (ADVI), Incremental VI (IVI), Non-conjugate Variational Message Passing (NC-VMP), Simplified NC-VMP (SNC-VMP), Conjugate-Computation VI (CCVI), Power EP (PEP), Alpha-divergence PEP (ADPEP), Convergent Power EP (CPEP), Stochastic”
Variational Heteroscedastic Gaussian Process Regression (2011, methodology): “In this work, we introduce a novel variational inference method for HGP regression that is based on variational Bayes and the Gaussian approximation (Opper & Archambeau, 2009; Nickisch & Rasmussen, 2008; Honkela et al., 2011).”
Doubly Stochastic Variational Bayes for non-Conjugate Inference (2014, methodology): “The maximisation relies on analytical integration (or in the worst case in one-dimensional numerical integration) for computing ∫ N (θ|μ,Σ) log g(θ)dθ, which subsequently can allow to tune the variational parameters (μ,Σ) using gradient optimization methods (Opper & Archambeau, 2009; Honkela et al.,”
Fast Variational Inference in the Conjugate Exponential Family (2012, methodology): “Honkela et al. [2010] applied conjugate gradients to the standard mean field bound, we expect much faster convergence for the KLC bound due to its differing curvature.”