TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

By Pengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian Riedel. 2020

818 citations recorded by the source corpus.

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “Structured knowledge encoding methods have been explored in many prior studies (Bogin et al., 2019; Lin et al., 2019; Yin et al., 2020a; Herzig et al., 2020; Agarwal et al., 2020; Saxena et al., 2020; Yasunaga and Liang, 2020; Yasunaga et al., 2021; Oguz et al., 2021).”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, neutral): “Unlike prior work where semi-structured data was used for reasoning over tables or knowledge-bases (Eisenschlos et al., 2020; Yin et al., 2020; Herzig et al., 2020; Yu et al., 2021), here we harness tables to allow LMs to reason over text directly.”
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, methodology): “One of the most representative is BERT [10], which has been widely used in Text-to-SQL and achieved SOTA performances at that time [4, 52].”
MATE: Multi-view Attention for Table Transformer Efficiency (2021, methodology): “For WIKITQ we close the gap with the best published system TABERT (Yin et al., 2020) (51.”
TABBIE: Pretrained Representations of Tabular Data (2021, methodology): “More recently, these pretraining methods have been extended to jointly learn representations of tables as well as text (Herzig et al., 2020; Yin et al., 2020), which enables improved modeling of tasks such as question answering over tables.”
TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021, background): “…like question answering (Dua et al., 2019; Ran et al., 2019; Andor et al., 2019; Chen et al., 2020a; Pasupat and Liang, 2015; Herzig et al., 2020; Yin et al., 2020; Zhang and Balog, 2020) and arithmetic word problems (Kush-man et al., 2014; Mitra and Baral, 2016; Huang et al., 2017; Ling et al.,…”
Language Models are Realistic Tabular Data Generators (2022, methodology): “In the light of these successes, transformer-based models have also been devised for tabular data classiﬁcation (Arik & Pﬁster, 2019; Somepalli et al., 2021; Kossen et al., 2021) and learning joint representations of tabular and textual data (Yin et al., 2020).”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, neutral): “Test (Liang et al., 2018) 42.3 43.1 (Dasigi et al., 2019) 42.1 43.9 (Agarwal et al., 2019) 43.2 44.1 (Herzig et al., 2020b) - 48.8 (Yin et al., 2020b) 52.2 51.8 (Wang et al., 2019) 43.7 44.5 w. RoBERTa-large 50.7(+7.0) 50.9(+6.4) w.”
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2020, background): “, 2020a) and new pretraining techniques (Yin et al., 2020; Herzig et al., 2020; Yu et al., 2020; Deng et al., 2020; Shi et al., 2021).”
Large Language Models are few(1)-shot Table Reasoners (2022, methodology): “In this paper, we mainly consider TAPAS (Herzig et al., 2020), TABERT (Yin et al., 2020), and TAPEX (Liu et al., 2021).”
Knowledge Graphs 2021: A Data Odyssey (2021, methodology): “In fact, similar approaches have lately been leveraged for DB problems, like QA over DB tables [13, 36], QA over web tables and text [40], or entity matching for data integration [1, 16].”
Language Models are Few-shot Learners for Prognostic Prediction (2023, background): “Recently, there have been a few trials (Yin et al., 2020; Bertsi-mas et al., 2022) investigating various serialization techniques and exploring the corresponding performance across different tasks, which turns out that LLMs for tabular modeling rely more on the correct values than the structure of…”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “First, these methods are highly flexible [25], allow for efficient and iterative training, and are particularly valuable for AutoML [26]–[31].”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, neutral): “Yin et al. (2020) use a language model for semantic parsing of natural language queries over tabular data.”
Deep entity matching with pre-trained language models (2020, background): “For numeric data, a good candidate solution would be a hybrid neural network similar to [20, 62] that combines the numeric features with the textual features.”
SpreadsheetCoder: Formula Prediction from Semi-structured Context (2021, methodology): “First, these models are designed for question answering (Herzig et al., 2020; Yin et al., 2020) or fact veriﬁcation (Chen et al., 2020), thus their inputs are the concatenation of a natural language question/statement and a table.”