Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions

By Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin, Mark W. Schmidt, Masashi Sugiyama. 2015

58 citations recorded by the source corpus.

Several recent works have explored stochastic gradient methods for variational inference that exploit the geometry of the variational-parameter space. However, the theoretical properties of these methods are not well-understood and these methods typically only apply to conditionally-conjugate models. We present a new stochastic method for variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-form updates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space. Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectives by using a more general class of divergence functions. Beyond giving a theoretical justification for a variety of recent methods, our experiments show that new algorithms derived in this framework lead to state of the art results on a variety of problems. Further, due to its generality, we expect that our theoretical analysis could also apply to other applications.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Composing graphical models with neural networks for structured representations and fast inference (2016, background): “[24] Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin, Mark Schmidt, and Masashi Sugiyama.”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, background): “Natural gradients are also better suited for VI because they can improve convergence rates by exploiting the information geometry of posterior approximations (Khan et al., 2016).”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, background): “It is also possible to define optimization approaches analogous to the above that do not linearize the free-energy term and instead perform potentially multiple updates of the nested non-linear optimization problems [Theis and Hoffman, 2015, Khan et al., 2015, 2016].”
Boosting Variational Inference: an Optimization Perspective (2017, methodology): “Similarly, the sublinear convergence analysis of variational inference of [10] only holds where the ratio q/p”