Kernel methods in machine learning

By Thomas Hofmann, Bernhard Schölkopf, Alex Smola. 2007

2,234 citations recorded by the source corpus.

We review machine learning methods employing positive definite kernels. These methods formulate learning and estimation problems in a reproducing kernel Hilbert space (RKHS) of functions defined on the data domain, expanded in terms of a kernel. Working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions. The latter include nonlinear functions as well as functions defined on nonvectorial data. We cover a wide range of methods, ranging from binary classifiers to sophisticated methods for estimation with structured data.

Subfields: Artificial Intelligence, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, background): “Hofmann et al. (2008) also provides a general review of kernel methods in machine learning.”
Prototypal Analysis and Prototypal Regression (2017, methodology): “Kernel methods and reproducing kernel Hilbert space (RKHS) are widely used in machine learning to extend algorithms where only inner products among data points are required (Schölkopf and Smola, 2002; Shawe-Taylor and Cristianini, 2004; Hofmann et al., 2008).”
Computing functions of random variables via reproducing kernel Hilbert space representations (2015, neutral): “…or Kernel PCA, are based on mapping the data into a reproducing kernel Hilbert space (RKHS) H (Boser et al. 1992; Schölkopf and Smola 2002; Shawe-Taylor and Cris-tianini 2004; Hofmann et al. 2008; Steinwart and Christmann 2008), where the feature map (or kernel map ) Φ satisﬁes for all x , x (cid”
Filtering with State-Observation Examples via Kernel Monte Carlo Filter (2013, background): “he ‘ 1 norm. Notably, kernel methods allow X to be a set of structured data, such as images, texts or graphs. In fact, there exist various positive deﬁnite kernels developed for such structured data (Hofmann et al., 2008). Note that the notion of positive deﬁnite kernels is different from smoothing ”
On a Link Between Kernel Mean Maps and Fraunhofer Diffraction, with an Application to Super-Resolution Beyond the Diffraction Limit (2013, background): “This kind of interpretation can be beneficial also for other systems, and indeed it is suggested by the view of kernels as Green’s functions [16, 24]: the kernel k can be viewed as the Green’s function of P ∗P , where P is a regularization operator such that the RKHS norm can be written as ‖f‖k = ‖P”
Kernel Bayes' Rule (2010, positive): “ul tools for generalizing linear statistical approaches to nonlinear settings, through an embedding of the sample to a high dimensional feature space, namely a reproducing kernel Hilbert space (RKHS) [18, 28]. Examples include support vector machines, kernel PCA, and kernel CCA, among others. In the”
On the Schoenberg Transformations in Data Analysis: Theory and Illustrations (2010, neutral): “3 Some properties of the Schoenberg transformations”
Computational Optimal Transport (2018, methodology): “AAA9mnictVttcxu3EYbTl8Tum9N+bCdzreJO0lE0kuyZdCajmVgvlhQrlmxSshPT1vDlRJ994tE8UpbN8Ft/Tb+23/s7+g/aT/0L3V0ABxyJuwVUVzeScCCeZxd7wGIXOHaGaZKPV1f/ee2DH/34Jz/98KPrN37281/88lc3P/71SZ5NRt34uJul2ehJp53HaTKIj8fJOI2fDEdx+7yTxo87r7bw88cX8ShPskFz/HYYPztv9wfJWdJtj6Hq9OYnLeJ4Oup3nk1Xbi+vLq98OYtavfh0enl6e3Z6c2l1ZZV+”
Random Feature Attention (2021, methodology): “Using the well-established random feature maps (Rahimi & Recht, 2007; Avron et al., 2016; §2), R FA approximates the dot-then-exponentiate function with a kernel trick (Hofmann et al., 2008): exp( x · y ) ≈ φ ( x ) · φ ( y ) .”
Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo (2015, background): “Exponential families (in Hilbert space) are an example, see e.g. Bialek et al. (2001); Hofmann et al. (2008); Wainwright & Jordan (2008).”
Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data (2020, background): “stic regression provides interpretable coefficients but SVM with an RBF kernel transforms the features to a high-dimensional space; this transformation cannot be easily retrieved since it is implicit [15]. Moreover, the PCA features are not easily interpretable. Therefore, we remove a feature set on”
Cooling Strategies for the Moment-Generating Function in Bayesian Global Optimization (2018, background): “Briefly, in Gaussian process regression, the correlation between the function values at different points are assumed to be jointly Gaussian and their covariance Cov{f(x), f(x′)} is specified by the kernel function [8]: k ∶ D × D → R.”
Generalization Error of Invariant Classifiers (2016, background): “Moreover, GE bounds can lead to a principled derivation of learning algorithms with GE guarantees, e.g., Support Vector Machine (SVM) (Cortes and Vapnik, 1995) and its extension to non-linear classiﬁcation with kernel machines (Hofmann et al., 2008).”
Explainable Machine Learning for Scientific Insights and Discoveries (2019, methodology): “Namely, it is the unique solution of a convex optimization problem that can be reproducibly obtained [12], [13], resulting in algorithmic transparency.”