Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages

By Wittawat Jitkrittum, Arthur Gretton, Nicolas Manfred Otto Heess, S. M. Ali Eslami, Balaji Lakshminarayanan, Dino Sejdinovic, Zoltán Szabó. 2015

33 citations recorded by the source corpus.

We propose an efficient nonparametric strategy for learning a message operator in expectation propagation (EP), which takes as input the set of incoming messages to a factor node, and produces an outgoing message as output. This learned operator replaces the multivariate integral required in classical EP, which may not have an analytic expression. We use kernel-based regression, which is trained on a set of probability distributions representing the incoming messages, and the associated outgoing messages. The kernel approach has two main advantages: first, it is fast, as it is implemented using a novel two-layer random feature representation of the input message distributions; second, it has principled uncertainty estimates, and can be cheaply updated online, meaning it can request and incorporate new training data when it encounters inputs on which it is uncertain. In experiments, our approach is able to solve learning problems where a single message operator is required for multiple, substantially different data sets (logistic regression for a variety of classification problems), where it is essential to accurately assess uncertainty and to efficiently and robustly update the message operator.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Discriminative Embeddings of Latent Variable Models for Structured Data (2016, neutral): “, 2010; 2011) and kernel EP (Jitkrittum et al., 2015), our current work exploits feature space embedding to reformulate graphical model inference procedures.”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Inspired by the work of Eslami et al. (2014), Jitkrittum et al. (2015) proposes Just-In-Time kernel regression for learning to pass expectation propagation (EP) messages.”
Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server (2015, methodology): “In addition, Heess et al. (2013), Eslami et al. (2014), Jitkrittum et al. (2015) use learning techniques to speed-up the process by directly predicting the natural parameters given properties of the tilted distribution.”
Proposal : Scalable , Active and Flexible Learning on Distributions (2015, unknown): “Jitkrittum et al. (2015) trained mmd-based gp regression for the messages computed by numerical integration in an expectation propagation system, and saw substantial speedups by doing so. We, inspired by J. B. Oliva, Neiswanger, et al. (2014), consider a problem where we not only obtain speedups ove”
Stochastic Expectation Propagation (2015, background): “At first sight, it therefore appears well suited to large-data problems: the locality of computation make the algorithm simple to parallelise and distribute, and good practical performance on a range of small data applications suggest that it will be accurate [9, 10, 11].”
Linear-Time Learning on Distributions with Approximate Kernel Embeddings (2015, methodology): “Expensive expectation propagation messages can be sped up by learning a “just-in-time” regression model (Jitkrittum et al., 2015).”
Meta-Learning an Inference Algorithm for Probabilistic Programs (2021, methodology): “For a message-passing-style variational-inference algorithm, such as expectation propagation (Minka, 2001; Wainwright & Jordan, 2008), Jitkrittum et al. (2015) studied the problem of learning a mechanism to pass a message for a given single inference task.”
Mondrian Forests for Large-Scale Regression when Uncertainty Matters (2015, methodology): “We use a simplified version of the dataset described in [15], where the goal is to predict the outgoing message in expectation propagation (EP) from a set of incoming messages.”