Stochastic Expectation Propagation

By Yingzhen Li, José Miguel Hernández-Lobato, Richard E. Turner. 2015

121 citations recorded by the source corpus.

Expectation propagation (EP) is a deterministic approximation algorithm that is often used to perform approximate Bayesian parameter learning. EP approximates the full intractable posterior distribution through a set of local approximations that are iteratively refined for each datapoint. EP can offer analytic and computational advantages over other approximations, such as Variational Inference (VI), and is the method of choice for a number of models. The local nature of EP appears to make it an ideal candidate for performing Bayesian learning on large models in large-scale dataset settings. However, EP has a crucial limitation in this context: the number of approximating factors needs to increase with the number of data-points, N, which often entails a prohibitively large memory overhead. This paper presents an extension to EP, called stochastic expectation propagation (SEP), that maintains a global posterior approximation (like VI) but updates it in a local way (like EP). Experiments on a number of canonical learning problems using synthetic and real-world datasets indicate that SEP performs almost as well as full EP, but reduces the memory consumption by a factor of N. SEP is therefore ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Progress & Compress: A scalable framework for continual learning (2018, methodology): “We will instead take a stochastic EP (Li et al., 2015) approach, which does not keep explicit approximations for each factor.”
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, background): “The recently proposed stochastic EP (Li et al., 2015) and BB-α (Hern ´ andez-Lobato et al., 2016) inference methods reduce this memory overhead to O (1) by sharing these approximating factors.”
Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (2016, positive): “We also ﬁnd that our results are comparable with the more recent results reported on the same datasets [e.g., 32–34] which leverage some advanced techniques that we can also beneﬁt from.”
Deep Gaussian Processes for Regression using Approximate Expectation Propagation (2016, methodology): “Approximations of this form were recently used in the SEP algorithm (Li et al., 2015) and although seemingly limited, in practice were found to perform almost as well as full EP while significantly reducing EP’s memory requirement, from O(NLM(2)) to O(LM(2)) in our case.”
Black-box $\alpha$-divergence Minimization (2015, methodology): “However, the energy function of SEP/AEP is unknown, making the attempts for proving convergence of these algorithms difﬁcult.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “…(Blundell et al., 2015; Graves, 2011; Louizos and Welling, 2016), assumed density ﬁltering (Hern´andez-Lobato and Adams, 2015), expectation propagation (Hasenclever et al., 2015; Li et al., 2015) or stochastic gradient Langevin diffusion methods (Korattikara et al., 2015; Welling and Teh, 2011).”
Variational Implicit Processes (2018, background): “, 1999) and expectation propagation (EP)-like algorithms (Minka, 2001; Li et al., 2015).”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, background): “A.1 B AYESIAN S AMPLING Bayesian sampling has been a long-standing tool in Bayesian modeling, with a wide range of applications such as uncertainty modeling (Li et al., 2015), data generation (Feng et al., 2017; Chen et al., 2018a) and reinforcement learning (Zhang et al., 2018).”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “Li et al. [2015] show that their stochastic power EP algorithm recovers precisely these same updates when α→ 0 (this is related to property 6).”
Advances in Variational Inference (2017, background): “Some recent extensions of EP [101], [125], [204], [228] can be viewed as classical EP with”
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, methodology): “A few uncertainty modelling methods have been proposed based on Bayesian neural network [19, 16, 10, 14, 26].”
Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data (2014, methodology): “Since its introduction by Opper and Winther (2000) and Minka (2001b), EP has been an important Bayesian computational method for inferring intractable posterior densities.”
Rényi Divergence Variational Inference (2016, methodology): “Also these two proposals can be viewed as extending recent attempts of distributed posterior approximation (by carving up the dataset into pieces with M > 1 [9, 10]) to both SEP and BB-α.”
Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server (2015, background): “…µ>i θ′i +A(θ′i) )) subject to µ0 ∈M [µi, νi] ∈Mi for i = 1, . . . , n θ′i ∈ Θ for i = 1, . . . , n µ0 = µi for i = 1, . . . , n (31) Maximizing over θ′i while keeping the other variables fixed will simply set θ ′ i = ∇A∗(µi), so that the KL terms vanish and resulting in the original problem (23).”