An empirical comparison of supervised learning algorithms

By R. Caruana, Alexandru Niculescu-Mizil. 2006

2,894 citations recorded by the source corpus.

A number of supervised learning methods have been introduced in the last decade. Unfortunately, the last comprehensive empirical evaluation of supervised learning was the Statlog Project in the early 90's. We present a large-scale empirical comparison between ten supervised learning methods: SVMs, neural nets, logistic regression, naive bayes, memory-based learning, random forests, decision trees, bagged trees, boosted trees, and boosted stumps. We also examine the effect that calibrating the models via Platt Scaling and Isotonic Regression has on their performance. An important aspect of our study is the use of a variety of performance criteria to evaluate the learning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Interpreting Models via Single Tree Approximation (2016, methodology): “Random forests [Breiman, 2001] and other machine learning methods have been widely used across different disciplines and are acknowledged for their outstanding predictive power [Caruana and Niculescu-Mizil, 2006].”
Learning Optimized Risk Scores (2016, neutral): “[22] Caruana, Rich and Alexandru Niculescu-Mizil.”
Benefits of Depth in Neural Networks (2016, background): “Lastly, decision trees and boosted decision trees are practically successful classes usually viewed as competitors to neural networks (Caruana and Niculescu-Mizil, 2006), and have the following structure. and thereafter conditionally evaluates either a left l -dt or a right r -dt where l + r < k .”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “Empirical benchmarks have long contributed to the scientific understanding, advancement, and validation of machine learning techniques [8, 6, 7, 46, 14, 13].”
Pose Machines: Articulated Pose Estimation via Inference Machines (2014, methodology): “In this work, we use a boosted classifier [28] with random forests for the weak learners, because random forests have been empirically shown to consistently outperform other methods on several datasets [29].”
Machine learning–XGBoost analysis of language networks to classify patients with epilepsy (2017, background): “They are significantly better classifiers than other algorithms (see [38]).”
Empirical methods for the estimation of Southern Ocean CO 2 : support vector and random forest regression (2017, positive): “As with SVM, random forests (both classification and regression variants) have also been used in remote sensing applications, though it does not seem to be as widely used in earth system sciences despite proving to be a powerful, yet easy to implement, learning algorithm (Caruana and Niculescu-Mizil”
Selecting Near-Optimal Learners via Incremental Data Allocation (2015, background): “In such a scenario, the practitioner may inevitably resort to the traditional approach to ﬁnding the best learner [cf. 12], namely, brute-force training of all learners on the full training set, and selecting the one with best validation accuracy.”
Recommendation systems: Principles, methods and evaluation (2015, background): “If trained on very high quality data, they have the ability to make very accurate predictions [56].”
MediBoost: a Patient Stratification Tool for Interpretable Decision Making in the Era of Precision Medicine (2016, methodology): “Although we present MediBoost algorithms with only two branches per node in this paper (stumps) for simplicity, it could easily be extended to multi-branch nodes, where each node will represent a tree similar to those used in boosted trees with the corresponding improvement in accuracy as shown by C”
Forecasting cancellation rates for services booking revenue management using data mining (2010, background): “The literature on multi-class probability estimation (Caruana and Niculescu-Mizil 2006) is rich.”
Which method predicts recidivism best?: a comparison of statistical, machine learning and data mining predictive models (2013, background): “There have been many attempts, across a multitude of disciplines and situations, to establish which model performs best (King et al. , 2006; Lim et al. , 1998, 2000; Caruana and Niculescu-Mizil, 2006).”
Survey of computational intelligence as basis to big flood management: challenges, research directions and future work (2018, result): “Developing a flexible, real-time method by considering fuzzy logic’s concept, Mendel (1995) classified debris flow warnings into several levels in the framework of an EWS.”
Application of machine learning to construction injury prediction (2016, result): “This is in accordance with Caruana and Niculescu-Mizil [14] who compared boosted trees and RF for 11 binary classi ﬁ cation problems and found that boosted trees have a slight edge over RF (performance was evaluated based on 8 different metrics).”
Machine Learning in Radiation Oncology: Opportunities, Requirements, and Needs (2018, methodology): “If accuracy is favored over interpretability, then tree base methods such as random forests or gradient boosting, and Support Vector Machines with kernel methods, consistently win most modeling competitions when structured data are analyzed (such as the type of data described above) (25, 26).”
Robust Brain Extraction Across Datasets and Comparison With Publicly Available Methods (2011, methodology): “1) Random Forests: The discriminative model in this study is a voxel-based Random Forest [21] classifier, which has been proven successful in a variety of domains and compares favorably with other state-of-the-art algorithms [25].”