Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis

By Tony Van Gestel, Johan A. K. Suykens, Gert R. G. Lanckriet, Annemie Lambrechts, B. De Moor, Joos Vandewalle. Neural Computation, 2002

311 citations recorded by the source corpus.

The Bayesian evidence framework has been successfully applied to the design of multilayer perceptrons (MLPs) in the work of MacKay. Nevertheless, the training of MLPs suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units. In support vector machines (SVMs) for classification, as introduced by Vapnik, a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high-dimensional kernel-induced feature space in which a linear large margin classifier is constructed. Practical expressions are formulated in the dual space in terms of the related kernel function, and the solution follows from a (convex) quadratic programming (QP) problem. In least-squares SVMs (LS-SVMs), the SVM problem formulation is modified by introducing a least-squares cost function and equality instead of inequality constraints, and the solution follows from a linear system in the dual space. Implicitly, the least-squares formulation corresponds to a regression formulation and is also related to kernel Fisher discriminant analysis. The least-squares regression formulation has advantages for deriving analytic expressions in a Bayesian evidence framework, in contrast to the classification formulations used, for example, in gaussian processes (GPs). The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression. In this article, the Bayesian evidence frame-work is combined with the LS-SVM classifier formulation. Starting from the feature space formulation, analytic expressions are obtained in the dual space on the different levels of Bayesian inference, while posterior class probabilities are obtained by marginalizing over the model param-eters. Empirical results obtained on 10 public domain data sets show that the LS-SVM classifier designed within the Bayesian evidence framework consistently yields good generalization performances.

Subfields: Analytical Chemistry, Computer Vision and Pattern Recognition, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/55f42d7f12818780817c5bd231c020ba75fe592d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

Hybrid and ensemble-based soft computing techniques in bankruptcy prediction: a survey (2010, methodology): “Van Gestel (2006), in contrast, find hyper-parameters for the least squares support vector machine (LS-SVM) by applying the Bayesian evidence framework (MacKay 1992; Gestel et al. 2002).”
Benchmarking Least Squares Support Vector Machine Classifiers (2004, methodology): “…Sequential Minimal Optimization (SMO) (Platt, 1998) and iteratively reweighted least squares approaches (IRWLS) (Navia-Vázquez et al., 2001), while LS-SVMs (Suykens & Vandewalle, 1999b; Suykens et al., 2002; Van Gestel et al., 2001, 2002; Viaene et al., 2001) result into a set of linear equations.”
Kernel Methods and Machine Learning (2014, background): “Learning models of this type have been explored by numerous researchers [10, 63, 76, 147, 178, 179, 180, 181, 189, 226, 241].”
KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition (2005, methodology): “Subsequent research saw the development of a series of KFD algorithms (see Baudat and Anouar [6], Roth and Steinhage [7], Mika et al. [8], [9], [10], Yang [11], Lu et al. [12], Xu et al. [13], Billings and Lee [14], Gestel et al. [15], Cawley and Talbot [16], and Lawrence and Schö lkopf [17]).”
Classifying Compound Structures in Satellite Images: A Compressed Representation for Fast Queries (2015, methodology): “Thus, these normalized word frequencies associated to the region labels are ingest in a machine learning system to train a linear classiﬁer (linear discriminant analysis (LDA) [51] or SVM [52]), which provides the weight vector ¯ ω and a threshold ¯ τ .”
A new fuzzy support vector machine to evaluate credit risk (2005, methodology): “References [14], [26], and [32] provide a probability framework for understanding SVM or least square SVM.”
Multiobjective Intelligent Energy Management for a Microgrid (2013, methodology): “For a training data set D : {xn, yn}Nn=1, the optimal regularization parameter α b is calculated based on Bayesian inference [24]–[26].”