TABBIE: Pretrained Representations of Tabular Data

By Hiroshi Iida, Dung Ngoc Thai, Varun Manjunatha, Mohit Iyyer. 2021

228 citations recorded by the source corpus.

Existing work on tabular representation-learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table-based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model’s learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, background): “More recently, self-supervised pre-training has also shown promising results on modalities other than plain text, such as tables (Herzig et al., 2020; Deng et al., 2020; Iida et al., 2021), knowledge bases (Zhang et al., 2019; Peters et al., 2019) and image-text (Su et al., 2020).”
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022, background): “, 2020a), TABBIE (Iida et al., 2021), TabGCN (Pramanick and Bhattacharya, 2021) and RCI (Glass et al.”
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (2021, unknown): “TABBIE [20] is an adaptation of axial attention that applies self-attention to rows and columns separately, then averages the representations and passes them as input to the next layer.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Recent works [127, 128, 78] pretrain LMs with large tabular datasets, improving the question answering systems by reasoning over the tables.”
A Cookbook of Self-Supervised Learning (2023, methodology): “Furthermore, it is common with tabular data to predict mask vectors as a pretext task [Yoon et al., 2020, Iida et al., 2021].”