XGBoost: A Scalable Tree Boosting System
By Tianqi Chen, Carlos Guestrin. 2016
57,634 citations recorded by the source corpus.
Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.
Subfields: Artificial Intelligence, Optimization & Training, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
In Optimization & Training
- Frank Hutter
- Yoshua Bengio
- Katharina Eggensperger
- Matthias Feurer
- Quoc V. Le
- Stefan Falkner
- Martin Wistuba
- Holger H. Hoos
In ML Systems & Hardware
- Frank Hutter
- Tong Wang
- Sebastian Raschka
- Su-In Lee
- Scott M. Lundberg
- Alex Smola
- Edouard Oyallon
- Satoshi Hara
Citations in the corpus
- SecureBoost: A Lossless Federated Learning Framework (2019, methodology): “To construct the tree with lower communication cost, we take advantage of an approximate framework proposed by (Chen and Guestrin 2016), where the detailed calculation is shown in Algorithm 1.”
- On the Art and Science of Machine Learning Explanations (2018, background): “Monotonic XGBoost is an excellent option to evaluate because the software is open source, readily available, easily installable and de-ployable, and highly scalable [7].”
- Consistent Individualized Feature Attribution for Tree Ensembles (2018, methodology): “Tree ensemble implementations in popular packages such as XGBoost [6], scikit-learn [20], and the gbm R package [22] allow a user to compute a measure of feature importance.”
- Making Tree Ensembles Interpretable (2016, methodology): “As model P, we used XGBoost (Chen & Guestrin, 2016).”
- Fair Regression: Quantitative Definitions and Reduction-based Algorithms (2019, methodology): “We considered two variants of LS and LR oracles: linear learners from scikit-learn (Pedregosa et al., 2011), and tree ensembles from XGBoost (Chen & Guestrin, 2016).”
- NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “We split corresponding datasets into equal training and testing parts and used XGBoost [8] for both problems.”
- Survey on Automated Machine Learning (2019, methodology): “Besides the possibility to tune scikit-learn estimators, auto ml provides interfaces to other popular ML libraries like TensorFlow (Abadi et al., 2016) or XGBoost (Chen and Guestrin, 2016).”
- The hardness of conditional independence testing and the generalised covariance measure (2018, methodology): “In particular, we use the R package xgboost [12, 13] with a ten-fold cross-validation scheme over the parameter maxdepth.”
- Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning (2021, methodology): “We compare our model with three sets of baselines: (1) interpretable baselines (π-score of one) such as linear models and generalized additive models, (2) partially interpretable baseline, EBM (π-score less than one), and (3) a black-box baseline (π-score of zero), XGBoost.”
- Mathematical optimization in classification and regression trees (2021, background): “One of the most popular implementations is XGBoost (Chen and Guestrin 2016), which is praised as highly accurate and scalable.”
- Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020, positive): “We use the XGBoost implementation [6].”
- The age of secrecy and unfairness in recidivism prediction (2018, methodology): “To then investigate whether the COMPAS general score depends in a more complex way on the Criminal Involvement Subscale items listed in the Appendix in Table 12, we ran several machine learning algorithms (random forests (31), boosted decision trees (32), and support vector machines with a radial ba”
- FFORMA: Feature-based forecast model averaging (2020, methodology): “We used the gradient tree boosting model of xgboost as the underlying implementation of the learning model (Chen & Guestrin 2016).”
- Quantitative missense variant effect prediction using large-scale mutagenesis data (2017, positive): “Instead, during training, the algorithm uses variants with missing features to determine how feature missingness should handled by the model at each tree node (Chen and Guestrin, 2016).”
- FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “In recent years, gradient boosting [13] based algorithms [2,7,23] achieve excellent performance on various tasks.”