Why do tree-based models still outperform deep learning on typical tabular data?
By Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux. Neural Information Processing Systems, 2022
1,782 citations recorded by the source corpus.
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ( ∼ 10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs). This leads to a series of challenges which should guide researchers aiming to build tabular-specific NNs: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner. Results Looking at the results as a function of random search time rather than random search iterations tree-based models superiority even more striking. Neural networks and tree-based models were close for some benchmarks after a small number of iterations, but for the same amount of time spent on random search, tree-based models scores are always high above neural networks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences
View Paper: https://www.semanticscholar.org/paper/5a00b32876f7d4869bce980500d4ccc978389315
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Longo Luca
- Sujay Khandagale
- Ben Feuer
- Micah Goldblum
- Ganesh Ramakrishnan
- C. VishakPrasad
- Jonathan Valverde
- Chinmay Hegde
In Computer Vision and Pattern Recognition
Citations in the corpus
- Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, methodology): “Here, traditional ML techniques are often adopted, especially techniques based on Decision Trees (DTs) [63].”
- A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “When it comes to adapting to tabular data, deep learning models have generally underperformed compared to tree-based models such as XGBoost and random forests [399, 400].”
- TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, neutral): “However, several recent comprehensive reviews (Shwartz-Ziv and Armon, 2022; Borisov et al., 2022a; Grinsztajn et al., 2022) found that gradient-boosted tree ensembles like XGBoost (Chen and Guestrin, 2016) and LightGBM (Ke et al.”
- When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “…advances in neural network (NN) architectures for tabular data [4, 47], there is still an active debate over whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with multiple works arguing either for [4, 37, 40, 47, 51] or against [7, 26, 27, 55] NNs.”
- Accurate predictions on small data with a tabular foundation model (2025, neutral): “14, there are 22 tabular classification datasets selected based on criteria such as heterogeneous columns, moderate dimensionality and sufficient difficulty.”
- A Fair Benchmarking of Deep Relational Database Learning Models (2026, unknown): “Deep learning for tabular data has long struggled to demonstrate robust performance compared to traditional machine learning methods [9, 10].”