Revisiting Deep Learning Models for Tabular Data

By Yu. V. Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko. Neural Information Processing Systems, 2021

1,551 citations recorded by the source corpus.

The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems. In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5fa06d856ba6ae9cd1366888f8134d7fd0db75b9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “The regression datasets used in these experiments have been used in recent studies of machine learning with tabular data [26, 34, 47, 53]; each dataset corresponds to an OpenML task, and can be preprocessed exactly like the classification datasets used in other experiments.”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “Moreover, several recent works have shown that the proposed sophisticated architectures are not superior to properly tuned simple models, like MLP and ResNet [13, 18].”
ADBench: Anomaly Detection Benchmark (2022, methodology): “Supervised ResNet and FTTransformer tailored for tabular data have been open-sourced in their original paper [56].”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, methodology): “To model the reverse process, we use a simple MLP architecture adapted from (Gorishniy et al., 2021): As in (Nichol, 2021; Dhariwal & Nichol, 2021), a tabular input x in , a timestep t , and a class label y are processed as follows: t emb = Linear ( SiLU ( Linear ( SinTimeEmb ( t )))) where…”
Accurate predictions on small data with a tabular foundation model (2025, positive): “Although transformer-based models can be applied to tabular data 26,27 , TabPFN addresses two key limitations inherent to them.”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Performancewise, multiple independent evaluations demonstrate that deep neural network methods from the hybrid (Section IV-B1) and transformer-based (Section IV-B2) groups exhibit superior predictive performance compared to plain deep neural networks on various datasets [9], [48], [62], [84].”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, positive): “…deep learning publications claim to beat or match tree-based models, but their claims have been put into question: a simple Resnet seems to be competitive with some of these new models [Gorishniy et al., 2021], and most of these methods seem to fail on new datasets [Shwartz-Ziv and Armon, 2021].”
Hopular: Modern Hopfield Networks for Tabular Data (2022, background): “The FT-Transformer maps features to tokens that are fed into a transformer (Gorishniy et al., 2021).”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Recent approaches have shown potential for resolving problems of older approaches, even if more research must be performed to confirm this [72,73].”
Why do tree-based models still outperform deep learning on tabular data? (2022, positive): “We benchmark the following deep models: MLP : a classical MLP from Gorishniy et al. [2021].”
Multi-modal Deep Learning (2024, methodology): “At the beginning I applied the vanilla transformer [24],tab-transformer [25] and FT-transformer [26].”
A Fair Benchmarking of Deep Relational Database Learning Models (2026, unknown): “The tabular learning community has benefited from extensive benchmarking studies [5, 6, 7, 8].”
NeurIDA: Dynamic Modeling for Effective In-Database Analytics (2025, unknown): “Implementation details could be found in https://github.com/nusdbsystem/NeurIDA.”
Griffin: Towards a Graph-Centric Relational Database Foundation Model (2025, unknown): “More recently, transformer-based methods like TabTransformer (Huang et al., 2020), TabNet (Arik & Pfister, 2021), FT-Transformer (Gorishniy et al., 2021), and SAINT (Somepalli et al., 2021) have lever-aged attention mechanisms to capture complex relationships within rows and columns.”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “GradNorm RF (Breiman, 2001), CatBoost (Prokhorenkova et al., 2018), and LightGBM (Ke et al., 2017); Tabular Foundation Models (TFM), including TabPFN (Hollmann et al., 2023) and TabICL (Qu et al., 2025); Deep Tabular Models (DTM), including DNN (LeCun et al., 2015), DeepFM (Guo et al., 2017), FTTran”