Boosting the margin: A new explanation for the effectiveness of voting methods

By Robert E. Schapire, Yoav Freund, Peter Barlett, Wee Sun Lee. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, background): “Examples of such models include logistic regression on linearly-separable data, non-parametric regression (Liang and Rakhlin, 2018; Belkin et al., 2019), boosting (Schapire et al., 1998), and over-parameterized deep neural networks (Zhang et al., 2017).”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “Related ideas have been considered in the context of margins theory (7, 9, 10), where a larger function class H may permit the discovery of a classifier with a larger margin.”
Random Forests (2001, methodology): “The synthetic data was described in Breiman (1996) and also used in Schapire et al. (1997). There were 50 runs.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “…numbers, algorithmic stability, Rademacher complexity, or flat minima (see, for instance, Srebro et al., 2010; Kakade et al., 2009; Zhou, 2002; Schapire et al., 1998; Koltchinskii et al., 2002; Bousquet and Elisseeff, 2002; Vapnik and Chervonenkis, 1971); the size of the Rashomon set differs…”
Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron (2018, background): “[30] Robert E Schapire, Yoav Freund, Peter Bartlett, Wee Sun Lee, et al.”
Learning privately from multiparty data (2016, background): “Advantages of ensemble approaches in general have been analyzed previously, in terms of bias-variance decomposition (Breiman, 1996b), and in terms of the margin of training samples (Schapire et al., 1998).”
Causal inference by using invariant prediction: identification and confidence intervals (2015, background): “For example, ˆ B could be the set of variables with non-zero regression coeﬃcient with square-root Lasso estimation [Belloni et al., 2011], Lasso [Tibshirani, 1996] or boosting [Schapire et al., 1998, Friedman, 2001, B¨uhlmann and Yu, 2003] with cross-validated penalty parameter.”
Natural Language Processing for EHR-Based Computational Phenotyping (2018, background): “the margin between positive and negative examples [125].”
Rademacher and Gaussian Complexities: Risk Bounds and Structural Results (2003, background): “…a number of results bounding error in terms of a sample average of a margin error plus a penalty term involving the complexity of the real-valued class (such as covering numbers and fat-shattering dimensions; see Bartlett, 1998, Mason et al., 2000, Schapire et al., 1998, Shawe-Taylor et al., 1998).”
Spectrally-normalized margin bounds for neural networks (2017, methodology): “This tool has been used to study the behavior of 2-layer networks, boosting methods, SVMs, and many others (Bartlett, 1996; Schapire et al., 1997; Boucheron et al., 2005); in boosting, for instance, there is a similar growth in complexity over time (each training iteration adds a weak learner),…”
The Implicit Bias of Gradient Descent on Separable Data (2017, background): “…t ρ ( t ) = ˜ w , where ˜ w is a solution to ∀ n ∈ S : η (cid:0) (cid:1) Analogies with Boosting Perhaps most similar to our study is the line of work on understanding AdaBoost in terms its implicit bias toward large L 1 -margin solutions, starting with the seminal work of Schapire et al. (1998).”
A Confidence-Based Approach for Balancing Fairness and Accuracy (2016, methodology): “The theory of margins has provided a deep, foundational explanation for the generalization properties of algorithms such as AdaBoost and soft-margin SVMs [2, 15].”
Newer Classification and Regression Tree Techniques: Bagging and Random Forests for Ecological Prediction (2006, methodology): “A related technique called ''boosting'' (Freund 1995; Schapire and others 1998) has recently gained popularity.”