Kernels for Vector-Valued Functions: a Review

By Mauricio A Álvarez, Lorenzo Rosasco, Neil D. Lawrence. 2011

1,083 citations recorded by the source corpus.

Kernel methods are among the most popular techniques in machine learning. From a regularization perspective they play a central role in regularization theory as they provide a natural choice for the hypotheses space and the regularization functional through the notion of reproducing kernel Hilbert spaces. From a probabilistic perspective they are the key in the context of Gaussian processes, where the kernel function is known as the covariance function. Traditionally, kernel methods have been used in supervised learning problems with scalar outputs and indeed there has been a considerable amount of work devoted to designing and learning kernels. More recently there has been an increasing interest in methods that deal with multiple outputs, motivated partially by frameworks like multitask learning. In this monograph, we review different methods to design or learn valid kernel functions for multiple outputs, paying particular attention to the connection between probabilistic and functional methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, background): “A review of kernels for vectorvalued functions can be found in Álvarez et al. (2012). Hofmann et al.”
Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages (2015, methodology): “Treating all outputs jointly can be achieved with a multi-output kernel (Alvarez et al., 2011).”
Learning Theory for Distribution Regression (2014, background): “, 2015); for a recent review on multi-output prediction methods see (Álvarez et al., 2011; Borchani et al., 2015).”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, background): “l function is deﬁned by the dot product of the neural network feature vector. By assuming for now that all possible tasks are simultaneously present similarly to multi-task GPs (Bonilla et al., 2008; Álvarez et al., 2012), the joint distribution over function values and output data for all tasks is ”
Continual Multi-task Gaussian Processes (2019, methodology): “These models generalise the ﬂexible prediction system of GP approaches to the vector-valued random ﬁeld setup (Alvarez et al., 2012).”
Efficient Output Kernel Learning for Multiple Tasks (2015, background): “The multi-task kernel on input and output is assumed to be decoupled as the product of a scalar kernel and the output kernel, which is a positive semidefinite matrix [1, 13, 14, 15].”
A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design (2017, methodology): “…through cross-covariance functions built using linear model of coregionalization or convolved process (Bonilla et al. 2007; Alvarez and Lawrence 2009; Alvarez et al. 2012), and the general stacked single-target framework and ensemble of regressor chains (Spyromitros-Xioufis et al. 2016).”
Exact Bayesian inference in spatiotemporal Cox processes driven by multivariate Gaussian processes (2015, background): “A review by Alvarez et al. (2012) discusses the relationship between co-regionalization and multiple-output models.”
Machine Learning and Materials Informatics: Recent Applications and Prospects (2017, background): “Such algorithms are indeed available [111,112], and if exploited can lead to prediction schemes of the electronic structure for new configurations of atoms.”
A Consistent Regularization Approach for Structured Prediction (2016, background): “This allows to define a (least squares) surrogate problem of the original structured one, that is cast within a multi-output regularized learning framework [9, 10, 11].”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “The GP is a non-parametric statistical model which has been extensively used in various scenarios, e.g., active learning [8], multi-task learning [9], [10], manifold learning [11], and optimization [12].”
Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes (2017, methodology): “To that end, we construct a linear model of coregionalization (LMC) [15], which mixes two intrinsic coregionalization models as follows where k w ( x, x (cid:48) ) , w ∈ { 0 , 1 } , is the radial basis function (RBF) with automatic relevance determination, i.e. ,w , . . . , 2 d,w ) , with d,w being…”
Computationally Efficient Convolved Multiple Output Gaussian Processes (2011, background): “For systems involving high amounts of coupled differential equations (Álvarez et al., 2009;́Alvarez et al., 2011a; Honkela et al., 2010), the approach presented here is a reasonable way of obtaining approximate solutions and incorporating prior domain knowledge to the model.”
Accurate interatomic force fields via machine learning with covariant kernels (2016, background): “Here, we show how Vectorial Gaussian Process (VGP) [29, 30] regression provides a more natural framework for force learning, where the correct vector behaviour of forces under symmetry transformations can be obtained by using a new family of vector kernels of co-variant nature.”