Learning Nonlinear Functions Using Regularized Greedy Forest

By Rie Johnson, Tong Zhang. 2011

180 citations recorded by the source corpus.

We consider the problem of learning a forest of nonlinear decision rules with general loss functions. The standard methods employ boosted decision trees such as Adaboost for exponential loss and Friedman's gradient boosting for general loss. In contrast to these traditional boosting algorithms that treat a tree learner as a black box, the method we propose directly learns decision forests via fully-corrective regularized greedy search using the underlying forest structure. Our method achieves higher accuracy and smaller models than gradient boosting on many of the datasets we have tested on.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Wang
2. Yufei Xia
3. Chuanzhe Liu
4. Yuying Li
5. Nana Liu
6. Jingyi Yang
7. Yunyi Li
8. Boxiang Wang

Citations in the corpus

• Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning (2021, background): “(10) The above tree-structured regularization has been introduced by (Johnson & Zhang, 2013) to prevent the boosting machines from overfitting.”
• XGBoost: A Scalable Tree Boosting System (2016, methodology): “This this resembles previous work on regularized greedy forest [22], but simplifies the objective and algorithm for parallelization.”
• Scene Classification via a Gradient Boosting Random Convolutional Network Framework (2016, background): “During the last few years, many empirical studies have been published, which use decision trees as the base learners for GBM [15], [16].”
• A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring (2017, methodology): “The regularized objective function, which is inspired by 3 http://scikit-learn.org/stable/modules/ensemble.html#gradient-tree-boosting AC CE PT ED M AN US CR IP T the regularized greedy forest (Johnson & Zhang, 2014), tends to smooth the contributions of base learners to avoid over-fitting.”
• The Higgs boson machine learning challenge (2014, methodology): “It is based on the Regularized Greedy Forest (RGF) algorithm (Johnson and Zhang, 2014), a variation on gradient boosting that decouples structure search and optimization.”
• Gradient boosting machines, a tutorial (2013, background): “Moreover the GBMs have shown considerable success in not only practical applications, but also in various machine-learning and data-mining challenges (Bissacco et al., 2007; Hutchinson et al., 2011; Pittman and Brown, 2011; Johnson and Zhang, 2012).”
• Ensemble boosted trees with synthetic features generation in application to bankruptcy prediction (2016, methodology): “In this paper we propose a novel method for bankruptcy prediction that makes use of Extreme Gradient Boosting (Chen and He, 2015b) for developing regularized boosted trees (Johnson and Zhang, 2011; Chen and He, 2015a).”