Distributed Gaussian Processes

By Marc Peter Deisenroth, Jun Wei Ng. 2015

390 citations recorded by the source corpus.

To scale Gaussian processes (GPs) to large data sets we introduce the robust Bayesian Committee Machine (rBCM), a practical and scalable product-of-experts model for large-scale distributed GP regression. Unlike state-of-theart sparse GP approximations, the rBCM is conceptually simple and does not rely on inducing or variational parameters. The key idea is to recursively distribute computations to independent computational units and, subsequently, recombine them to form an overall result. Efficient closed-form inference allows for straightforward parallelisation and distributed computations with a small memory footprint. The rBCM is independent of the computational graph and can be used on heterogeneous computing infrastructures, ranging from laptops to clusters. With sufficient computing resources our distributed GP model can handle arbitrarily large data sets.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “We consider the flight delay data set (Hensman et al., 2013; Deisenroth and Ng, 2015; Lakshminarayanan et al., 2016), a large scale regression benchmark with several published results.”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, methodology): “…low-rank kernel matrix approximation [74]; (2) local structures of the kernel matrix [75]; (3) the state-space model reformulation and Kalman ﬁlter [41]; (4) the Bayesian committee machine (BCM) with a number of distributed computing units [76]; and (5) the variational Bayesian formulation [77].”
Mondrian Forests for Large-Scale Regression when Uncertainty Matters (2015, methodology): “Deisenroth and Ng [6] present the robust Bayesian committee machine (rBCM), which combines predictions from experts that operate on subsets of data.”
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization (2017, background): “For our experiments, we set βi = 1 M + 1 ∀i = 1, . . . , M + 1, (38) which does not influence the predicted mean as the terms cancel out; however, this effectively increases the uncertainty which the general model of experts usually tends to underestimate (Deisenroth and Ng 2015).”
Hybrid Models with Deep and Invertible Features (2019, methodology): “Following Deisenroth & Ng (2015), we train using the ﬁrst 5 million data points and use the following 100 , 000 as test data.”
Noise Contrastive Priors for Functional Uncertainty (2018, result): “We consider the flight delay data set (Hensman et al., 2013; Deisenroth and Ng, 2015; Lakshminarayanan et al., 2016), a large scale regression benchmark with several published results.”
A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Restrictions apply. been extensively employed in a range of early proposals [15], [16], [17], [18], [19], [20], [21], [22], [23], [59].”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “The NPAE provides theoretically consistent predictions at the cost of requiring much higher time complexity due to the inversion of a new M × M kernel matrix at each test point. retaining To be efﬁcient while consistent predictions, instead of using the ﬁxed GP prior to correct the aggregation like ”
Black-box data-efficient policy search for robotics (2017, background): “Possible solutions include using local GPs [39], [40] or to stop using GPs and make use of recent advances in neural networks with uncertain predictions [41], [42].”
Residual Mixture of Experts (2022, neutral): “Various of expert architectures are proposed [10, 12, 43, 48, 51].”
Blitzkriging: Kronecker-structured Stochastic Gaussian Processes (2015, background): “These subsets reduce the expressivity of the model, however Deisenroth and Ng [6] have had success scaling these models to large data sets.”
Learning Scalable Deep Kernels with Recurrent Structure (2016, methodology): “Finally, the semi-stochastic learning procedure naturally complements research in asynchronous optimization (e.g., Deisenroth and Ng, 2015).”
Statistical Model Aggregation via Parameter Matching (2019, neutral): “These approaches roughly fall into two groups, those [9, 15, 21, 22, 23] that decompose a global, centralized learning objective into localized ones that can be optimized separately using local data, and those that iterate between training local models on private data sources and distilling them int”
FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply (2020, methodology): “Substantial research has focused on developing gate function models, such as SVMs [27, 56], Gaussian Process [28, 55], and neutral networks [47, 46].”