TabTransformer: Tabular Data Modeling Using Contextual Embeddings

By Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar S. Karnin. arXiv.org, 2020

875 citations recorded by the source corpus.

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a2ec47b9bcc95d2456a8a42199233e5d9129ef18

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-modal Deep Learning (2024, methodology): “At the beginning I applied the vanilla transformer [24],tab-transformer [25] and FT-transformer [26].”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “Despite a large number of proposed architectures for tabular DL [2, 3, 13, 17, 21, 24, 31, 39, 40], the performance gap between them and the “shallow” ensembles of decision trees, like GBDT, often remains significant [13, 36].”
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (2021, positive): “Transformer models for more general tabular data include TabTransformer [18], which uses a transformer encoder to learn contextual embeddings only on categorical features.”
Hopular: Modern Hopfield Networks for Tabular Data (2022, background): “The TabTransformer learns contextual embeddings of categorical features (Huang et al., 2020).”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “Architecture-based methods design specialized neural architectures for tabular data [11, 28, 47], a large sub-class of which are transformer-based architectures [4, 26, 34, 56].”
Why do tree-based models still outperform deep learning on tabular data? (2022, neutral): “…Machines Guo et al. [2017], tabular-speciﬁc transformers architectures Somepalli et al. [2021], Kossen et al. [2021], Arik and Pﬁster [2019], Huang et al. [2020], and various regularization techniques to adapt classical architectures to tabular data [Lounici et al., 2021, Shavitt and Segal,…”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “In this domain, information is mainly presented as tabular or temporal data.”
Revisiting Deep Learning Models for Tabular Data (2021, background): “Due to the ubiquitous success of attention-based architectures for different domains (Dosovitskiy et al., 2021; Vaswani et al., 2017), several authors propose to employ attention-like modules for tabular DL as well (Arik and Pfister, 2020; Huang et al., 2020a; Song et al., 2019).”
Language Models are Few-shot Learners for Prognostic Prediction (2023, background): “Many attempts to apply transformers on tabular modeling ( e.g. , TabTrans-former Huang et al., 2020) have also achieved success.”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “TabTransformer [99] X Transformer network for categorical data”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, methodology): “…Machines Guo et al. [2017], tabular-specific transformers architectures Somepalli et al. [2021], Kossen et al. [2021], Arik and Pfister [2019], Huang et al. [2020], and various regularization techniques to adapt classical architectures to tabular data [Lounici et al., 2021, Shavitt and Segal,…”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “We consider two baselines here: TabNet [158] and TabTransformer [157].”
Tabular Data: Deep Learning is Not All You Need (2021, methodology): “Recent works have suggested both inter-sample attention, in which features of a given sample interact with each other, and intra-sample attention, in which data points interact with each other using entire rows/samples [Arik and Pﬁster, 2021, Huang et al., 2020, Somepalli et al., 2021].”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, neutral): “Self-supervised objectives have largely revolved around the prediction of masked cells, the iden-tiﬁcation or correction of corrupted cells, and contrastive losses over augmentations (Bahri et al., 2022; Somepalli et al., 2021; Yoon et al., 2020; Arik and Pﬁster, 2021; Huang et al., 2020).”
Griffin: Towards a Graph-Centric Relational Database Foundation Model (2025, unknown): “More recently, transformer-based methods like TabTransformer (Huang et al., 2020), TabNet (Arik & Pfister, 2021), FT-Transformer (Gorishniy et al., 2021), and SAINT (Somepalli et al., 2021) have lever-aged attention mechanisms to capture complex relationships within rows and columns.”