SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

By Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, Tom Goldstein. arXiv.org, 2021

556 citations recorded by the source corpus.

Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5fa2103e36b3e76e49edb8433a1206a6b25e3ead

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Cookbook of Self-Supervised Learning (2023, methodology): “, 2018] in embedding space is also an effective augmentation for tabular data [Somepalli et al., 2021].”
Hopular: Modern Hopfield Networks for Tabular Data (2022, methodology): “We extend this selection by two datasets for regression: (a) colleges was already used for other Deep Learning methods for tabular data (Somepalli et al., 2021), and (b) sulfur is publicly available and fits with its 10,082 instances well into the existing collection of medium-sized datasets.”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “Architecture-based methods design specialized neural architectures for tabular data [11, 28, 47], a large sub-class of which are transformer-based architectures [4, 26, 34, 56].”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, methodology): “While many recent works try to 24 address this problem with a native DL approach for tabular classification [2, 43, 19, 22], we go one 25 step further and simply apply a transformer [46] that itself learned to do classification, completely 26 bypassing the question of how to train a DL model on a ta”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[159] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein.”
Tabular Data: Deep Learning is Not All You Need (2021, background): “Recent works have suggested both inter-sample attention, in which features of a given sample interact with each other, and intra-sample attention, in which data points interact with each other using entire rows/samples [Arik and Pﬁster, 2021, Huang et al., 2020, Somepalli et al., 2021].”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, methodology): “SAINT : a Transformer model with an embedding module and an inter-samples attention mechanism, proposed in Somepalli et al. [2021].”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Self-attention and intersample attention transformer (SAINT) [9] is a hybrid attention approach, combining self-attention [5] with intersample attention over multiple rows.”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “SAINT : a Transformer model with an embedding module and an inter-samples attention mechanism, proposed in Somepalli et al. [2021].”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, unknown): “Self-supervised objectives have largely revolved around the prediction of masked cells, the iden-tiﬁcation or correction of corrupted cells, and contrastive losses over augmentations (Bahri et al., 2022; Somepalli et al., 2021; Yoon et al., 2020; Arik and Pﬁster, 2021; Huang et al., 2020).”
Language Models are Realistic Tabular Data Generators (2022, methodology): “In the light of these successes, transformer-based models have also been devised for tabular data classiﬁcation (Arik & Pﬁster, 2019; Somepalli et al., 2021; Kossen et al., 2021) and learning joint representations of tabular and textual data (Yin et al., 2020).”
Griffin: Towards a Graph-Centric Relational Database Foundation Model (2025, unknown): “More recently, transformer-based methods like TabTransformer (Huang et al., 2020), TabNet (Arik & Pfister, 2021), FT-Transformer (Gorishniy et al., 2021), and SAINT (Somepalli et al., 2021) have lever-aged attention mechanisms to capture complex relationships within rows and columns.”