Statistical mechanics of Support Vector networks.

By R. Dietrich, Manfred Opper, Haim Sompolinsky. 1998

Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks. For nonlinear classification rules, the generalization error saturates on a plateau, when the number of examples is too small to properly estimate the coefficients of the nonlinear part. When trained on simple rules, we find that SVMs overfit only weakly. The performance of SVMs is strongly enhanced, when the distribution of the inputs has a gap in feature space.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “In addition, we calculate the average (cid:104) Z n D under a replica symmetric ansätz and a Gaussian approximation by following the calculation procedure of the previous works (Dietrich et al., 1999; Bordelon et al., 2020; Canatar et al., 2021).”
Data mining in clinical big data: the frequently used databases, steps, and methodological models (2021, methodology): “SVMs The SVM is a relatively new classification or prediction method developed by Cortes and Vapnik and represents a data-driven approach that does not require assumptions about data distribution [52].”
Basic Tenets of Classification Algorithms K-Nearest-Neighbor, Support Vector Machine, Random Forest and Neural Network: A Review (2020, methodology): “The SVM is an extension of the support vector classifier and is obtained as a result from the enlargement of the feature space in a specific way, using kernels [29].”
Geometric compression of invariant manifolds in neural nets (2020, methodology): “In the physics literature, β has been computed in regression or classification tasks for fixed kernels [25, 16, 26, 27, 28].”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, methodology): “In statistical physics domain, (Dietrich et al., 1999) calculated learning curves for support vector machines, but not kernel regression, in the limit of number of training samples going to infinity for dot product kernels with binary inputs using a replica method.”
Advances in Large Margin Classifiers (2000, unknown): “A somewhat more detailed analysis which was designed for readers with a Statistical Physics background, can be found in [Dietrich et al., 1999].”
Statistical mechanics of complex neural systems and high dimensional data (2013, background): “Statistical mechanics has also had success in the analysis of learning in other architectures and machine learning algorithms, including support vector machines [150]–[152] and Gaussian processes [153].”
Statistical Mechanics of Learning : Generalization (2002, background): “An application to the generalization performance of supportvector machines can be found in (Dietrich et al, 1999).”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “A key step is a Gaussian approximation to the average over the dataset in the feature space [52], which exploits the orthogonality of the feature vectors with respect to the input distribution p(x).”
Estimating Dataset Size Requirements for Classifying DNA Microarray Data (2003, unknown): “It can be argued (Dietrich et al. , 1999) that a Support Vector Machine with linear kernel, used in our numerical simulations, has the same generalization error dependence as the optimal perceptron (Opper et al. , 1990).”