Student-t Processes as Alternatives to Gaussian Processes

By Amar Shah, Andrew Gordon Wilson, Zoubin Ghahramani. 2014

227 citations recorded by the source corpus.

We investigate the Student-t process as an alternative to the Gaussian process as a nonparametric prior over functions. We derive closed form expressions for the marginal likelihood and predictive distribution of a Student-t process, by integrating away an inverse Wishart process prior over the covariance kernel of a Gaussian process model. We show surprising equivalences between different hierarchical Gaussian process models leading to Student-t processes, and derive a new sampling scheme for the inverse Wishart process, which helps elucidate these equivalences. Overall, we show that a Student-t process can retain the attractive properties of a Gaussian process -- a nonparametric representation, analytic marginal and predictive distributions, and easy model selection through covariance kernels -- but has enhanced flexibility, and predictive covariances that, unlike a Gaussian process, explicitly depend on the values of training observations. We verify empirically that a Student-t process is especially useful in situations where there are changes in covariance structure, or in applications like Bayesian optimization, where accurate predictive covariances are critical for good performance. These advantages come at no additional computational cost over Gaussian processes.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, methodology): “This was done numerically in [143] and analytically using conjugate priors in [138], Fig.”
ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language (2019, background): “For example, systems may have complex noise [42, 22], yield multiple types of observations [12], depend on covariates [25], have interrelated subsystems [47], and more.”
Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions (2015, methodology): “However, other models have shown to be useful in some Bayesian optimization tasks e.g. Student-t process priors [8] and deep neural networks [9].”
A new acquisition function for Bayesian optimization based on the moment-generating function (2017, background): “, Studentt processes [14], [15] and random forest [13].”
Variational Implicit Processes (2018, methodology): “3 Inverse Wishart process as a prior for kernel functions Definition 4 (Inverse Wishart processes (Shah et al., 2014)).”
Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models (2015, methodology): “ore direct dependency may be pre ferable. An option is to use Student's t processes instead where the posterior variance depends on t he observed function values through a global scaling factor (Shah et al., 2014). 27 G UTMANN AND C ORANDER 7. Applications We here apply the developed methodolog”
Functional Variational Bayesian Neural Networks (2019, methodology): “[27] Amar Shah, Andrew Wilson, and Zoubin Ghahramani.”
Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes (2017, methodology): “Most commonly used models are Gaussian processes (GPs) (Rasmussen and Williams, 2006), Random Forests (Thornton et al., 2013), T-Student processes (Shah et al., 2014) or Deep Neural Networks (Snoek et al., 2015).”