Benchmarking Least Squares Support Vector Machine Classifiers

By Tony Van Gestel, Johan A. K. Suykens, Bart Baesens, Stijn Viaene, Jan Vanthienen, Guido Dedene, B. De Moor, Joos Vandewalle. Machine-mediated learning, 2004

535 citations recorded by the source corpus.

In Support Vector Machines (SVMs), the solution of the classification problem is characterized by a (convex) quadratic programming (QP) problem. In a modified version of SVMs, called Least Squares SVM classifiers (LS-SVMs), a least squares cost function is proposed so as to obtain a linear set of equations in the dual space. While the SVM classifier has a large margin interpretation, the LS-SVM formulation is related in this paper to a ridge regression approach for classification with binary targets and to Fisher's linear discriminant analysis in the feature space. Multiclass categorization problems are represented by a set of binary classifiers using different output coding schemes. While regularization is used to control the effective number of parameters of the LS-SVM classifier, the sparseness property of SVMs is lost due to the choice of the 2-norm. Sparseness can be imposed in a second stage by gradually pruning the support value spectrum and optimizing the hyperparameters during the sparse approximation procedure. In this paper, twenty public domain benchmark datasets are used to evaluate the test set performance of LS-SVM classifiers with linear, polynomial and radial basis function (RBF) kernels. Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances. These SVM and LS-SVM performances are consistently very good when compared to a variety of methods described in the literature including decision tree based algorithms, statistical algorithms and instance based learning methods. We show on ten UCI datasets that the LS-SVM sparse approximation procedure can be successfully applied.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/37b0dcf6dbb57074e7e7f0f81ca684be0fe9533c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Empirical mode decomposition–based least squares support vector regression for foreign exchange rate forecasting (2012, methodology): “Therefore, least squares support vector regression (LSSVR), proposed by Suykens and Vandewalle (1999), has been applied in much literature as an alternative (He et al., 2010; Khemchandani et al., 2009); it is a simplified version of traditional SVR that alters inequality constraints into equal condi”
A Convex Formulation for Learning Task Relationships in Multi-Task Learning (2010, methodology): “When there is only one task and Ω is given as a positive scalar, our model will degenerate to the probabilistic model for regularized least-squares regression and least-squares SVM [13].”
Least squares linear discriminant analysis (2007, methodology): “One of the directions for future work is to compare this new formulation with other related algorithms for multi-class classiflcation algorithms [10, 11 , 27].”
Heterogeneous-Neighborhood-based Multi-Task Local Learning Algorithms (2013, background): “, square loss l(s, t) = (s − t) as in the least square SVM [10].”
Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings (2008, methodology): “It is known that a careful tuning of such hyperparameters is essential to obtain a representative assessment of the classifier’s potential (see, e.g., [3], [ 63 ]).”
Supervised classification and mathematical optimization (2013, methodology): “Among others, it is worthwhile mentioning the so-called Least Squares SVM proposed in [104, 221] and the Proximal Support Vector Machine in [98, 101].”
A Regularization Approach to Learning Task Relationships in Multitask Learning (2014, methodology): “When there is only one will degenerate to the probabilistic for regularized least-squares regression and least-squares SVM [Gestel et al. 2004].”
Auto-WEKA : combined selection and hyperparameter optimization of supervised machine learning algorithms (2014, background): “This can be partially addressed by starting out with a very coarse discretization, then refining the upper and lower bounds of hyperparameters to explore the area around the grid point with the best performance in the previous iteration [Van Gestel et al., 2004].”
Combing Random Forest and Least Square Support Vector Regression for Improving Extreme Rainfall Downscaling (2019, methodology): “In the training period of LS-SVR, the grid-search method [58] is used to estimate optimal parameters.”
Building Acceptable Classification Models (2010, background): “Benchmarking studies have shown that, in general, nonlinear models provide the most accurate predictions [6, 56], as they are able to capture nonlinearities in the”
Machine learning in side-channel analysis: a first study (2011, result): “LS-SVM classifiers have achieved considerable results in comparison to other learning techniques, including standard SVM classifiers, for classification on 20 public domain benchmark data sets [9].”