Tabular Data: Deep Learning is Not All You Need

By Ravid Shwartz-Ziv, Amitai Armon. Information Fusion, 2021

2,138 citations recorded by the source corpus.

A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/009560d2710138a446e6e254d8ddcb65eaa0e687

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “When it comes to adapting to tabular data, deep learning models have generally underperformed compared to tree-based models such as XGBoost and random forests [399, 400].”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, neutral): “…deep learning publications claim to beat or match tree-based models, but their claims have been put into question: a simple Resnet seems to be competitive with some of these new models [Gorishniy et al., 2021], and most of these methods seem to fail on new datasets [Shwartz-Ziv and Armon, 2021].”
Accurate predictions on small data with a tabular foundation model (2025, neutral): “Although standard neural networks, which unlike TabPFN do not use ICL, were shown to underperform for small ( 10,000 samples) tabular data 1,14,71 , as a point of reference, we still consider a simple neural network, the MLP. Tabular dataset benchmarks.”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “…advances in neural network (NN) architectures for tabular data [4, 47], there is still an active debate over whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with multiple works arguing either for [4, 37, 40, 47, 51] or against [7, 26, 27, 55] NNs.”
Why do tree-based models still outperform deep learning on tabular data? (2022, neutral): “, 2021], and most of these methods seem to fail on new datasets [Shwartz-Ziv and Armon, 2021].”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “This may be due to the heterogeneous nature of tabular data, with their features being sparse, type-mixed, and weaker in correlation than natural image-language data [152, 150].”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Although deep learning methods perform outstandingly well for classiﬁcation or data generation tasks on homogeneous data (e.g., image, audio, and text data), tabular data still pose a challenge to deep learning models [6], [7], [8].”
A Fair Benchmarking of Deep Relational Database Learning Models (2026, unknown): “The tabular learning community has benefited from extensive benchmarking studies [5, 6, 7, 8].”