Functional Gradient Techniques for Combining Hypotheses

By Llew Mason, Jonathan Baxter, Peter L. Bartlett, Marcus Frean. 2000

341 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/f5e23d650853dc7f3dbe4370d4ace6be55f931ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ENDER: a statistical framework for boosting decision rules (2010, methodology): “Two others, the gradient descent (Mason et al. 1999) and the gradient boosting (Friedman 2001), can be used with any differentiable loss function.”
A survey of appearance models in visual object tracking (2013, neutral): “In essence, this BDAM is an extension of the GradientBoost algorithm [Friedman 2001] and works similarly to the AnyBoost algorithm [Mason et al. 1999].”
Functional Frank-Wolfe Boosting for General Loss Functions (2015, methodology): “The hypothesis space being considered is the set of ensembles F of base hypothesis h j from the base hypothesis space H : Gradient boosting [2, 3] tries to ﬁnd an approximation ˆ F ( x ) that minimizes the empirical risk L ( F, y ) = 1 m =1 l ( F ( x i ) , y i ) on the training set.”
SMOTEBoost: Improving Prediction of the Minority Class in Boosting (2003, background): “Boosting has received extensive theoretical and empirical study [11, 12], but most of the published work focuses on”
Boosting with the L2-loss: regression and classification (2001, background): “…by National Science Foundation grants DMS-9803063 and FD01-12731 and Army Research Of ce grants DAAG5598-1-0341 and DAAD19-01-1-0643. has been the emergence of many variants of the original AdaBoost, under various loss or objective functions (Mason et al. 2000; Friedman et al. 2000; Friedman 2001).”
Empirical margin distributions and bounding the generalization error of combined classifiers (2002, methodology): “This also motivated the development of the methods of combining the classiﬁers based on explicit optimization of the penalized average cost function of the margins, see Mason, Bartlett and Baxter (1999), Mason, Baxter, Bartlett and Frean (1999).”
Learning to search: structured prediction techniques for imitation learning (2009, background): “Additionally, functional gradient descent algorithm have seen success in a number of real-world problems (Ratliff et al., 2006; Mason et al., 1999).”
Grafting: Fast, Incremental Feature Selection by Gradient Descent in Function Space (2003, background): “Discussion of this phenomenon, and many examples, can be found in Smola et al. (2000).”
SpeedBoost: Anytime Prediction with Uniform Near-Optimality (2012, background): “[6] and Friedman [4], one can view boosting algorithms as a modified version of gradient descent in the space of functions.”
Reducing multiclass to binary: a unifying approach for margin classifiers (2001, background): “It has been observed by Breiman (1997a, 1997b) and other authors (Collins, Schapire, & Singer, 2000; Friedman, Hastie, & Tibshirani, 2000; Mason, Baxter, Bartlett, & Frean, 1999; R¨ atsch, Onoda, & M¨ uller, to appear; Schapire & Singer, 1999) that the ’s and ’s are effectively being greedily chos”
Boosting with early stopping: Convergence and consistency (2005, methodology): “Previous analyses, including matching pursuit for least squares [29], Breiman’s analysis [9] of the exponential loss, as well as the Bregman divergence bound in [12] and the analysis of gradient boosting in [31], were all limiting results without any information on the numerical speed of convergence”
FloatBoost learning and statistical face detection (2004, background): “Several gradient boosting algorithms are proposed [7], [8], [9], which provide new insights into AdaBoost learning.”
Boosting Algorithms as Gradient Descent (1999, background): “The following theorem (proof omitted, see [15]) supplies a specific step-size for AnyBoost and characterizes the limiting behaviour with this step-size.”
Boosting and Maximum Likelihood for Exponential Models (2001, background): “Mason et al. [10] consider boosting algorithms as functional gradient descent and Duffy and Helmbold [5] study various loss functions with respect to the PAC boosting property.”
Managing Diversity in Regression Ensembles (2005, background): “The problem is somewhat easier if we have estimators that give posterior probabilities, in which case the effect ostimator correlations on classification error rate has been investigated by Tumer and Ghosh (1995) and Fumera and Roli (2003), though there remain several open questions on this topic.”
Robust Support Vector Machine Training via Convex Outlier Ablation (2006, methodology): “Unfortunately, even though robust hinge loss has played a significant role in generalization theory (Bartlett & Mendelson 2002; Shawe-Taylor & Cristianini 2004), the minimization objective (7) has not been often applied in practice because it is non-convex, and leads to significant difficulties in o”
On Early Stopping in Gradient Descent Learning (2007, background): “…been seen in [Rudin, Daubechies, and Schapire 2004] and on the second lead to the functional gradient descent view with general convex loss functions [e.g. Breiman 1999; Friedman, Hastie, and Tibshirani 2000; Mason, Baxter, Bartlett, and Frean 2000; Friedman 2001], where this paper also lies in.”