Why do tree-based models still outperform deep learning on tabular data?

By Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux. arXiv.org, 2022

618 citations recorded by the source corpus.

While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ($\sim$10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs). This leads to a series of challenges which should guide researchers aiming to build tabular-specific NNs: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/ef4a99f703bd6c51f86056313716c39ea48baeb8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, methodology): “The theoretical results by Ng (2004) and empirical results by Grinsztajn et al. (2022) imply, that TabPFN's diminishing performance when uninformative features are added is linked to it's relative rotation invariance.”
ADBench: Anomaly Detection Benchmark (2022, result): “The great performance of tree-based ensembles (in tabular AD) is consistent with the ﬁndings in literature [20, 58, 170], which may be credited to their capacity to handle imbalanced AD datasets via aggregation.”
A Fair Benchmarking of Deep Relational Database Learning Models (2026, unknown): “The tabular learning community has benefited from extensive benchmarking studies [5, 6, 7, 8].”