Boosting Algorithms as Gradient Descent

By Llew Mason, Jonathan Baxter, Peter L. Bartlett, Marcus Frean. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “[38] illustrates that gradient boosting algorithms can be viewed as iterative functional gradient descent algorithms.”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “Recall the AdaBoost algorithm which builds an ensemble of weak learners (they can be any learner in principle Mason et al. (1999)), each of which is ﬁtted upon iteratively re-weighted training data (Breiman, 1998).”
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification (2018, methodology): “At a technical level, our post-processing algorithm is similar to gradient boosting [10, 24].”
Rule Covering for Interpretation and Boosting (2020, background): “Gradient boosting algorithms use a sequence of weak learners and build a model iteratively [7, 13, 8].”
The Boosting Approach to Machine Learning An Overview (2003, background): “This view of boosting and its generalization are examined in considerable detail by Duffy and Helmbold [23], Mason et al. [51, 52] and Friedman [35].”
Learning Complexity-Aware Cascades for Deep Pedestrian Detection (2015, methodology): “using functional gradient descent on a classification risk [9, 20].”
Adapting boosting for information retrieval measures (2010, unknown): “MART is a boosted tree algorithm that performs gradient descent in function space [18].”
A Taxi Order Dispatch Model based On Combinatorial Optimization (2017, methodology): “In this work, we tried two popular models: linear logistic regression (LR) [5] and gradient boosted decision tree (GBDT) [6, 15].”
Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems (2019, background): “3 Brief Review of Functional Gradient Methods Functional gradient methods have been mainly studied for gradient boosting [22, 23] and kernel methods [24, 25, 26, 27, 28] in the machine learning community, but more recently it has been found to be useful in explaining the behavior of gradient descent”
Fast Binary Feature Selection with Conditional Mutual Information (2004, methodology): “…features according to their individual predictive power, which can be estimated by various means such as Fisher score (Furey et al., 2000), Kolmogorov-Smirnov test, Pearson correlation (Miyahara and Pazzani, 2000) or mutual information (Battiti, 1994; Bonnlander and Weigend, 1996; Torkkola, 2003).”
A Neural Support Vector Network architecture with adaptive kernels (2000, methodology): “[9] compare the performance of several voting methods that were shown to optimize a margin loss function.”
Twenty Years of Mixture of Experts (2012, background): “HME bears a resemblance to the boosting algorithm [107], [108] in that weak classifiers are combined to create a single strong classifier.”
Functional Gradient Boosting based on Residual Network Perception (2018, methodology): “For positive values γ , (cid:15) , p ≤ 2 , q , and K , a function √ This assumption is a counterpart of that imposed in Mason et al. (1999).”
Advance and Prospects of AdaBoost Algorithm (2013, unknown): “…xxx ) 表示加权条件期望 , 令 ω = (cid:36) ( xxx, y ) = e − yF ( xxx ) , 并定义 : 使用贪心策略以迭代方式在函数空间优化特定损失函数的分析理论不但有效地解释了 AdaBoost 算法的原理和泛化能力 , 同时也建立起一类 Ad-aBoost 变种算法设计的通用框架 [31] , 如 AnyBoost 算法优化任意可导损失函数 [32] , MarginBoost 算法优化基于分类间隔的损失函数 [32] , 文献 [33] 正是在 MarginBoost 算法框架下设计出的非对称代价 Boosting 算法 , 用于垃圾邮件检测 .”
Global Optimality in Neural Network Training (2017, unknown): “Nonetheless, by using arguments from gradient boosting, it is possible to show that problem (4) can be globally optimized by sequentially adding hidden units to the network until one can no longer find a hidden unit whose addition will decrease the objective function [3, 10, 17].”