Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

By Yarin Gal, Mark van der Wilk, Carl E. Rasmussen. 2014

154 citations recorded by the source corpus.

Gaussian processes (GPs) are a powerful tool for probabilistic inference over functions. They have been applied to both regression and non-linear dimensionality reduction, and offer desirable properties such as uncertainty estimates, robustness to over-fitting, and principled ways for tuning hyper-parameters. However the scalability of these models to big datasets remains an active topic of research. We introduce a novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm. This is done by exploiting the decoupling of the data given the inducing points to re-formulate the evidence lower bound in a Map-Reduce setting. We show that the inference scales well with data and computational resources, while preserving a balanced distribution of the load among the nodes. We further demonstrate the utility in scaling Gaussian processes to big data. We show that GP performance improves with increasing amounts of data in regression (on flight data with 2 million records) and latent variable modelling (on MNIST). The results show that GPs perform better than many common models often used for big data.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Stochastic Variational Deep Kernel Learning (2016, methodology): “On each dataset we used a fully-connected DNN which has the same architecture as in the airline delay task, except for DNN+ which has an additional hidden layer. as well as the distributed GP latent variable model (denoted as D-GPLVM) [9].”
Statistical Model Aggregation via Parameter Matching (2019, unknown): “These approaches roughly fall into two groups, those [9, 15, 21, 22, 23] that decompose a global, centralized learning objective into localized ones that can be optimized separately using local data, and those that iterate between training local models on private data sources and distilling them int”
Dropout as a Bayesian Approximation : Insights and Applications (2015, methodology): “The use of approximate variational inference for the model allows us to scale it to large data via stochastic and distributed inference (Hensman et al., 2013; Gal et al., 2014).”
Mathematics for Machine Learning (2020, background): “, 2015), or training of large-scale Gaussian process models (Hensman et al., 2013; Gal et al., 2014).”
Scalable Inference for Gaussian Process Models with Black-Box Likelihoods (2015, methodology): “Our method, unlike [9], uses a single discriminative multi-class framework.”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, methodology): “In order to make it adapt to the FedLoc framework, one may consider using the distributed variational inference techniques [135] with the GPSSM.”
Variational Auto-encoded Deep Gaussian Processes (2015, background): “Gal et al. (2014) and Dai et al. (2014) exploit a similar formulation for distributing the computation of BGPLVM, however, in their formulations, the gradients of variational parameters that depend on individual datapoints have to be collected centrally.”
Online Sparse Multi-Output Gaussian Process Regression and Learning (2019, methodology): “The variational formulation of sparse approximation was proposed in [39]–[43] and shown in [44] to outperform the FITC approximation.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “Alternatively, we can implement sparse approximations using advanced computing infrastructure, e.g., Graphics Processing Units (GPU) and distributed clusters/processors, to further speed up the computation.”
Stochastic Variational Inference for Gaussian Process Latent Variable Models using Back Constraints (2015, methodology): “Our work is an extension of [12] to GP models with latent variables, and a complementary perspective to a distributed training procedure introduced in [13], making further progress in scaling up inference and learning in GP models.”
Advances in Variational Inference (2017, background): “reparametrizations might be required to enable efficient distributed inference [49].”
Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs (2015, background): “Following the ideas of (Gal et al., 2014) we show that the approximation can be implemented in a distributed framework.”
Mondrian Forests for Large-Scale Regression when Uncertainty Matters (2015, methodology): “[10] use stochastic variational inference to speed up GPs, building on the variational bound developed by Titsias [28].”
Parametric Gaussian process regression for big data (2017, background): “resultingpredictionofthe modelis plottedin ﬁgure 1. 3 3.2 Airline delays The US ﬂight delay prediction example, originally proposed in [16], has reached a status of a standardbenchmarkdataset(seee.g.,[20,21,22,23,24])inGaussianprocessregression,partlybecause of the massive size of the dataset with n”
Distributed Gaussian Processes (2015, background): “Recently, Gal et al. (2014) proposed an approach that scales variational sparse GPs (Titsias, 2009) further by exploiting distributed computations.”