TabNet: Attentive Interpretable Tabular Learning

By Sercan Ö. Arik, Tomas Pfister. 2019

2,396 citations recorded by the source corpus.

We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet. TabNet uses sequential attention to choose which features to reason from at each decision step, enabling interpretability and more efficient learning as the learning capacity is used for the most salient features. We demonstrate that TabNet outperforms other variants on a wide range of non-performance-saturated tabular datasets and yields interpretable feature attributions plus insights into its global behavior. Finally, we demonstrate self-supervised learning for tabular data, significantly improving performance when unlabeled data is abundant.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2019, methodology): “ning state-ofthe-art performance on a variety of datasets. Time Series Interpretability with Attention: Attention mechanisms are used in translation [17], image classication [22] or tabular learning [23] to identify salient portions of input for each instance using the magnitude of attention weights”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, neutral): “TabNet is a widely used neural model for tabular data that uses attention over columns (Arik and Pﬁster, 2021).”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, neutral): “…al., 2018] or Factorization Machines Guo et al. [2017], tabular-specific transformers architectures Somepalli et al. [2021], Kossen et al. [2021], Arik and Pfister [2019], Huang et al. [2020], and various regularization techniques to adapt classical architectures to tabular data [Lounici et al.,…”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “Several recent works compare GBDTs to NNs on tabular data, often finding that either neural nets [4, 26, 37, 47] or GBDTs [7, 26, 27, 55] perform best.”
Language Models are Realistic Tabular Data Generators (2022, methodology): “In the light of these successes, transformer-based models have also been devised for tabular data classiﬁcation (Arik & Pﬁster, 2019; Somepalli et al., 2021; Kossen et al., 2021) and learning joint representations of tabular and textual data (Yin et al., 2020).”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “, image, audio, and text data), tabular data still pose a challenge to these models [5]–[7].”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, background): “Despite a large number of proposed architectures for tabular DL [2, 3, 13, 17, 21, 24, 31, 39, 40], the performance gap between them and the “shallow” ensembles of decision trees, like GBDT, often remains significant [13, 36].”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “Most importantly, MLPs do not match the performance of tree-based models such as GBDT on most of the datasets (Arik and Pfister 2019).”
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (2021, positive): “For example, TabNet [1] uses neural networks to mimic decision trees by placing importance on only a few features at each layer.”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “One such approach has integrated attention-based explanations into a neural architecture to achieve an efficient computation of tabular data and to increase its comprehensibility [74].”
VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain (2020, background): “The pretext task for self-supervised learning in TabNet [23] and TaBERT [24] is also recovering corrupted tabular data.”
Why do tree-based models still outperform deep learning on tabular data? (2022, neutral): “…et al., 2018] or Factorization Machines Guo et al. [2017], tabular-speciﬁc transformers architectures Somepalli et al. [2021], Kossen et al. [2021], Arik and Pﬁster [2019], Huang et al. [2020], and various regularization techniques to adapt classical architectures to tabular data [Lounici et al.,…”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, methodology): “Furthermore, DL methods such as TabNet, SAINT, Regularization Cocktails, Non-parametric Transformers (Arik and Pfister, 2021; Somepalli et al., 2021; Kadra et al., 2021; Kossen et al., 2021) are evaluated on much larger datasets and often use custom parameter tuning and preprocessing.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “We consider two baselines here: TabNet [158] and TabTransformer [157].”
Revisiting Deep Learning Models for Tabular Data (2021, background): “For these reasons, a large number of DL solutions were recently proposed, and new models continue to emerge (Arik and Pfister, 2020; Badirli et al., 2020; Hazimeh et al., 2020; Huang et al., 2020a; Klambauer et al., 2017; Popov et al., 2020; Song et al., 2019; Wang et al., 2017, 2020a).”