GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

By Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Y. Tan, Xinyi Yang, Dragomir R. Radev, Richard Socher, Caiming Xiong. 2020

296 citations recorded by the source corpus.

We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG) induced from existing text-to-SQL datasets. We pre-train our model on the synthetic data using a novel text-schema linking objective that predicts the syntactic role of a table field in the SQL for each question-SQL pair. To maintain the model's ability to represent real-world data, we also include masked language modeling (MLM) over several existing table-and-language datasets to regularize the pre-training process. On four popular fully supervised and weakly supervised table semantic parsing benchmarks, GraPPa significantly outperforms RoBERTa-large as the feature representation layers and establishes new state-of-the-art results on all of them.

Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Information Extraction

Citations in the corpus

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, neutral): “Unlike prior work where semi-structured data was used for reasoning over tables or knowledge-bases (Eisenschlos et al., 2020; Yin et al., 2020; Herzig et al., 2020; Yu et al., 2021), here we harness tables to allow LMs to reason over text directly.”
Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialogue State Tracking (2021, background): “Learning a structural inductive bias during pre-training has been shown beneﬁcial in downstream tasks that require parsing semantics, such as text-to-SQL (Yu et al., 2021) and table cell recognition (Wang et al., 2020).”
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2020, methodology): “Recently, meaningful advances emerged on the encoder side, including developments in Transformer-based architectures (Wang et al., 2020a) and new pretraining techniques (Yin et al., 2020; Herzig et al., 2020; Yu et al., 2020; Deng et al., 2020; Shi et al., 2021).”
TAPEX: Table Pre-training via Learning a Neural SQL Executor (2021, methodology): “to model structured data using pre-training techniques (Herzig et al., 2020; Yin et al., 2020; Yu et al., 2020; Wang et al., 2021b; Deng et al., 2020, 2021; Shi et al., 2021).”
Large Language Models are few(1)-shot Table Reasoners (2022, background): “…results reveal that LLMs (Ouyang et al., 2022; Chen et al., 2021a; Chowdhery et al., 2022) can achieve striking performance with only 1 or 2 demonstrations, e.g. 48.8% on WikiTable-Questions and 78.8% on TabFact, which are on par some near-SoTA models (Yu et al., 2021; Eisen-schlos et al., 2020).”
SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022, methodology): “Dataset To evaluate the performance of our proposed method, we conduct experiments on the Geo-Query dataset (Zelle and Mooney, 1996), where there are 880 queries to a database of U.S. geography.”
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, methodology): “For example, G RAPPA (Yu et al. 2021) and GAP (Shi et al. 2021) further pre-train the PLMs by using the schema item classiﬁcation task as one of the pre-training objectives.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, background): “, 2020) or derived automatically from existing data (Jia and Liang, 2016; Yu et al., 2020).”
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, methodology): “, 2020) and pretraining (Yin et al., 2020; Yu et al., 2021a) has led to parsers that already outperform Seq2Struct by more than 30% in parsing accuracy.”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, background): “Our results show that both generative and retrieval models can predict SQL queries that match the user’s intention, but the scores are much lower than the original text-to-SQL task (Finegan-Dollak et al., 2018; Dong and Lapata, 2018a; Yin and Neubig, 2018; Cao et al., 2019; Rubin and Berant, 2021; H”
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, background): “There is also work on tabular language models that encode both tables and text such as TaBERT (Yin et al., 2020), TaPas (Herzig et al., 2020), and Grappa (Yu et al., 2020).”
ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, background): “Meanwhile, there has also been work that uses pre-training to accommodate speciﬁc needs of downstream NLP tasks, such as open-domain retrieval (Guu et al., 2020), representing and predicting spans of text (Joshi et al., 2020) and semantic parsing (Yu et al., 2020; Deng et al., 2021).”