DART: Dropouts meet Multiple Additive Regression Trees
By Rashmi Vinayak, Ran Gilad-Bachrach. 2015
219 citations recorded by the source corpus.
Multiple Additive Regression Trees (MART), an ensemble model of boosted regression trees, is known to deliver high prediction accuracy for diverse tasks, and it is widely used in practice. However, it suffers an issue which we call over-specialization, wherein trees added at later iterations tend to impact the prediction of only a few instances, and make negligible contribution towards the remaining instances. This negatively affects the performance of the model on unseen data, and also makes the model over-sensitive to the contributions of the few, initially added tress. We show that the commonly used tool to address this issue, that of shrinkage, alleviates the problem only to a certain extent and the fundamental issue of over-specialization still remains. In this work, we explore a different approach to address the problem that of employing dropouts, a tool that has been recently proposed in the context of learning deep neural networks. We propose a novel way of employing dropouts in MART, resulting in the DART algorithm. We evaluate DART on ranking, regression and classification tasks, using large scale, publicly available datasets, and show that DART outperforms MART in each of the tasks, with a significant margin. We also show that DART overcomes the issue of over-specialization to a considerable extent.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- An overview of deep learning in medical imaging focusing on MRI (2018, neutral): “the DART technique for regression trees [63].”
- Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution (2018, unknown): “Rashmi and Gilad-Bachrach (2015) argued that GBT is sensitive towards the beginning, requiring lots of later trees to make an impact.”
- An Exploratory Analysis of Multi-Class Uncertainty Approximation in Bayesian Convolutional Neural Networks (2018, methodology): “For example, in [Rashmi and Gilad-Bachrach, 2015] dropout is used to increase the performance of an ensemble of boosted regression trees.”