Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server

By Leonard Hasenclever, Stefan Webb, Thibaut Lienart, Sandra Jeanne Vollmer, Balaji Lakshminarayanan, Charles Blundell, Yee Whye Teh. 2015

73 citations recorded by the source corpus.

This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not require any simplifying assumptions on the distribution of interest, beyond the existence of some Monte Carlo sampler for estimating the moments of the EP tilted distributions. Further, as opposed to EP which has no guarantee of convergence, SNEP can be shown to be convergent, even when using Monte Carlo moment estimates. Secondly, we propose a novel architecture for distributed Bayesian learning which we call the posterior server. The posterior server allows scalable and robust Bayesian learning in cases where a data set is stored in a distributed manner across a cluster, with each compute node containing a disjoint subset of data. An independent Monte Carlo sampler is run on each compute node, with direct access only to the local data subset, but which targets an approximation to the global posterior distribution given all data across the whole cluster. This is achieved by using a distributed asynchronous implementation of SNEP to pass messages across the cluster. We demonstrate SNEP and the posterior server on distributed Bayesian learning of logistic regression and neural networks. Keywords: Distributed Learning, Large Scale Learning, Deep Learning, Bayesian Learn- ing, Variational Inference, Expectation Propagation, Stochastic Approximation, Natural Gradient, Markov chain Monte Carlo, Parameter Server, Posterior Server.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, background): “Many other works have explored variety of approximation methods, e.g., Gal & Ghahramani (2016) use dropout for VI, Hernandez-Lobato & Adams (2015); Hasenclever et al. (2017) use expectation propagation, Li et al. (2016); Balan et al. (2015) use stochastic-gradient Langevin dynamics.”
Black-box $\alpha$-divergence Minimization (2015, background): “5), BB-only maintains the global parameter q and the prior parameter 0. This has been shown to achieve a signiﬁcant amount of memory saving. On the other hand, Gelman et al.,2014;Xu et al. ,2014;Teh et al. 2015), whether in a synchronous manner or not, extends to BB-naturally. But unlike EP, which”
Bayesian Recurrent Neural Networks (2017, background): “Hasenclever et al. (2015) derive a distributed expectation propagation scheme with SGLD (Welling & Teh, 2011) as an inner loop.”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, background): “…2016b]; local VI [Knowles and Minka, 2011, Wand, 2014, Khan and Lin, 2018]; power EP and related algorithms [Minka, 2001, 2004, Li et al., 2015, Hasenclever et al., 2017, Gelman et al., 2014]; and stochastic mini-batch variants of these algorithms [Hoffman et al., 2013, Li et al., 2015, Khan…”
Statistical Model Aggregation via Parameter Matching (2019, unknown): “These approaches roughly fall into two groups, those [9, 15, 21, 22, 23] that decompose a global, centralized learning objective into localized ones that can be optimized separately using local data, and those that iterate between training local models on private data sources and distilling them int”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “Hence, recent work on uncertainty in neural networks has focussed mostly on relatively faster approximate Bayesian solutions such as variational Bayesian methods [3, 14, 22], assumed density filtering [15], expectation propagation [21, 35] or stochastic gradient Langevin diffusion methods [18, 36].”
Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data (2014, background): “Extracting the core principles behind EP motivates a general framework for passing information between inferences on partitioned data (Xu et al., 2014; Hasenclever et al., 2015).”
Improving GAN Training via Binarized Representation Entropy (BRE) Regularization (2018, background): “However, computationally scalable deep Bayesian neural networks are still an active area of research (Hernández-Lobato & Adams, 2015; Hasenclever et al., 2017) and are not easy to use.”