Statistical Learning Theory
By Yuhai Wu. 2021
22,124 citations recorded by the source corpus.
A machine learning system, in general, learns from the environment, but statistical machine learning programs (systems) learn from the data. This chapter presents techniques for statistical machine learning using Support Vector Machines (SVM) to recognize the patterns and classify them, predicting structured objects using SVM, k-nearest neighbor method for classification, and Naive Bayes classifiers. The artificial neural networks are presented with brief introduction to error-correction rules, Boltzmann learning, Hebbian rule, competitive learning rule, and deep learning. The instance-based learning is treated in details with its algorithm and learning task. The chapter concludes with a summary, and a set of practice exercises.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Percy Liang
- Mohit Bansal
- Quoc V. Le
- Dacheng Tao
- Jianfeng Gao
- Ruslan Salakhutdinov
In Scaling & Learning Theory
- Vitaly Feldman
- Tengyu Ma
- Quanquan Gu
- Chiyuan Zhang
- Nati Srebro
- Simon Lacoste-Julien
- Percy Liang
- Naftali Tishby
Citations in the corpus
- mixup: Beyond Empirical Risk Minimization (2017, background): “Learning the function f by minimizing (1) is known as the Empirical Risk Minimization (ERM) principle (Vapnik, 1998).”
- Understanding deep learning requires rethinking generalization (2016, neutral): “These include VC dimension (Vapnik, 1998), Rademacher complexity (Bartlett & Mendelson, 2003), and uniform stability (Mukherjee et al., 2002; Bousquet & Elisseeff, 2002; Poggio et al., 2004).”
- The Graph Neural Network Model (2009, neutral): “Those techniques include random fields [42], Bayesian networks [43], statistical relational learning [44], transductive learning [45], and semisu-pervised approaches for graph processing [46].”
- On Calibration of Modern Neural Networks (2017, positive): “Learning theory suggests that regularization is necessary to prevent overfitting, especially as model capacity increases (Vapnik, 1998).”
- Deep learning via semi-supervised embedding (2008, methodology): “Transductive Support Vector Machines (TSVMs) (Vapnik, 1998) (which employs a kind of clustering) and LapSVM (Belkin et al., 2006) (which employs a kind of embedding) are examples of methods that are joint in their use of unlabeled data and labeled data, but their architecture is still shallow.”
- Optimization Methods for Large-Scale Machine Learning (2016, background): “respects the trade-os mentioned above. Structural Risk Minimization An approach for choosing a prediction function that has proved to be widely successful in practice is structural risk minimization [157,156]. Rather than choose a generic family of prediction functions|over which it would be both di”
- A theory of learning from different domains (2010, positive): “The uniform convergence bound on the α -error is nearly identical to the standard uniform convergence bound for hypothesis classes of finite VC dimension (Vapnik 1998; Anthony and Bartlett 1999), only with target and source errors weighted differently.”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, neutral): “learning principle [69], in which they go from a set examples to a general rule that works for any data coming from Fig.”
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, methodology): “While some such constraints can be efficiently embedded into the joint optimization process using the Lagrange Multipliers similar to those used in support vector machines [76], we found that the majority of them cannot be easily handled by the optimization algorithm.”
- Security analysis of online centroid anomaly detection (2010, neutral): “Starting from the mainstream computational learning theory (Vapnik, 1998; Schölkopf and Smola, 2002), a prevalent assumption is that training and test data are generated from the same, fixed but unknown, probability distribution.”
- Towards Causal Representation Learning (2021, background): “Such algorithms do exist, for instance, nearest neighbor classifiers, support vector machines, and neural networks [257, 217, 239, 66].”
- Causality for Machine Learning (2019, background): “So for a given sample size, it will depend on the problem being learned whether we achieve low expected error, and statistical learning theory provides probabilistic guarantees in terms of measures of complexity of function classes (Devroye et al., 1996; Vapnik, 1998).”
- Invariant Risk Minimization (2019, background): “Because most machine learning algorithms depend on the assumption that training and testing data are sampled independently from the same distribution [51], it is common practice to shuffle at random the training and testing examples.”
- Causal Regularization (2019, background): “To solve this task, regularization is known to be crucial for obtaining regression models that generalize well from training to test data [2].”
- Semi-supervised learning, causality, and the conditional cluster assumption (2019, methodology): “We compare with T-SVMs [8, 16] with linear and RBF kernels using the q3svm implementation [26], and with label propagation [14, 15] using the implementation in sci-kit learn [27].”
- Generalization in anti-causal learning (2018, background): “Consistency and convergence guarantees, for example within the empirical risk minimization (ERM) framework (Vapnik, 1998), paired with universal function approximation guarantees for neural nets (Hornik et al., 1989; Cybenko, 1989) prompt the conventional wisdom that “with enough data and powerful…”
- Strongly agree or strongly disagree?: Rating features in Support Vector Machines (2016, methodology): “A state-of-the-art method in Supervised Classi cation using a score function is the Support Vector Machines (SVM), Cristianini and Shawe-Taylor (2000), Vapnik (1995, 1998).”