A comparative analysis of gradient boosting algorithms

By Candice Bentéjac, Anna Csörgo, Gonzalo Martínez-Muñoz. Artificial Intelligence Review, 2019

2,475 citations recorded by the source corpus.

The family of gradient boosting algorithms has been recently extended with several interesting proposals (i.e. XGBoost, LightGBM and CatBoost) that focus on both speed and accuracy. XGBoost is a scalable ensemble technique that has demonstrated to be a reliable and efficient machine learning challenge solver. LightGBM is an accurate model focused on providing extremely fast training performance using selective sampling of high gradient instances. CatBoost modifies the computation of gradients to avoid the prediction shift in order to improve the accuracy of the model. This work proposes a practical analysis of how these novel variants of gradient boosting work in terms of training speed, generalization performance and hyper-parameter setup. In addition, a comprehensive comparison between XGBoost, LightGBM, CatBoost, random forests and gradient boosting has been performed using carefully tuned models as well as using their default settings. The results of this comparison indicate that CatBoost obtains the best results in generalization accuracy and AUC in the studied datasets although the differences are small. LightGBM is the fastest of all methods but not the most accurate. Finally, XGBoost places second both in accuracy and in training speed. Finally an extensive analysis of the effect of hyper-parameter tuning in XGBoost, LightGBM and CatBoost is carried out using two novel proposed tools.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7280df1bbbcaf743cb793d5bd654b4cf12b575f4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rohitash Chandra
2. Azal Ahmad Khan
3. Omkar Chaudhari

In Computer Vision and Pattern Recognition

1. Rohitash Chandra
2. Azal Ahmad Khan
3. Omkar Chaudhari

Citations in the corpus

• A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, methodology): “According to a study by Benntejac et al. [122], Gradient Boosting requires careful tuning of the parameters to achieve good performance.”