Functional Frank-Wolfe Boosting for General Loss Functions

By Chu Wang, Yingfei Wang, E. Weinan, Robert E. Schapire. 2015

21 citations recorded by the source corpus.

Boosting is a generic learning method for classification and regression. Yet, as the number of base hypotheses becomes larger, boosting can lead to a deterioration of test performance. Overfitting is an important and ubiquitous phenomenon, especially in regression settings. To avoid overfitting, we consider using $l_1$ regularization. We propose a novel Frank-Wolfe type boosting algorithm (FWBoost) applied to general loss functions. By using exponential loss, the FWBoost algorithm can be rewritten as a variant of AdaBoost for binary classification. FWBoost algorithms have exactly the same form as existing boosting methods, in terms of making calls to a base learning algorithm with different weights update. This direct connection between boosting and Frank-Wolfe yields a new algorithm that is as practical as existing boosting methods but with new guarantees and rates of convergence. Experimental results show that the test performance of FWBoost is not degraded with larger rounds in boosting, which is consistent with the theoretical analysis.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs (2019, methodology): “Our approach is inspired from a recent FW algorithm to solve l1Adaboost in the centralized and non-personalized setting (Wang et al., 2015).”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, methodology): “Therefore, other works consider functional Frank-Wolfe framework, where subproblems are linearized (Wang et al., 2015).”
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport (2018, background): “One thus generally resorts to space gridding (in low dimension) or to approximate steps, akin to boosting [36].”
MaxEntropy Pursuit Variational Inference (2019, methodology): “(1) is highly non-linear, we propose to follow the framework based on the Frank-Wolfe algorithm [26,17] and consider the constraint as a functional perturbation.”
Boosting Variational Inference: an Optimization Perspective (2017, methodology): “In the analysis of [24], the authors enforce a bounded polytope using functions in L1 with bounded L1 norm.”