When Gaussian Process Meets Big Data: A Review of Scalable GPs

By Haitao Liu, Yew-Soon Ong, Xiaobo Shen, Jianfei Cai. 2018

875 citations recorded by the source corpus.

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process regression (GPR), a well-known nonparametric, and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. However, they have not yet been comprehensively reviewed and analyzed to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this article is devoted to reviewing state-of-the-art scalable GPs involving two main categories: global approximations that distillate the entire data and local approximations that divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations that modify the prior but perform exact inference, posterior approximations that retain exact prior but perform approximate inference, and structured sparse approximations that exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “51 The uncertainty estimates are dependent both on the speciﬁcs of the approximations and on the prior probability distributions used.”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, background): “…the latter retrain part of it; (iii) Rehearsal based aims at preventing catastrophic forgetting by storing and replaying information from previous episodes (Rebuffi et al., 2017; Castro et al., 2018; Wu et al., 2018; Hou et al., 2019; Zhai et al., 2019; Liu et al., 2020b; jun Liu et al., 2020).”
Fast and Accurate Uncertainty Estimation in Chemical Machine Learning. (2018, methodology): “20,30 We use the Projected Process (PP) approximation, also called Projected Latent Variables, throughout this paper.”
Recent Advances in Bayesian Optimization (2022, background): “Although scalable GPs have been extensively studied in recent years to accommodate many observations [16, 112], these methods focus on the scenario where there exist a large amount of data while the dimension remains to be small or medium.”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, background): “A comprehensive survey of the existing scalable GP models can be found in [78].”