Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

By Peter Shaw, Ming-Wei Chang, Panupong Pasupat, Kristina Toutanova. 2020

197 citations recorded by the source corpus.

Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? To better assess this capability, we propose new train and test splits of non-synthetic datasets. We demonstrate that strong existing approaches do not perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a high-precision grammar-based approach with a pre-trained sequence-to-sequence model. It outperforms existing approaches across several compositional generalization challenges on non-synthetic data, while also being competitive with the state-of-the-art on standard evaluations. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, methodology): “In the prompt constructed to solve SCAN, we actually used both natural language and Python notation.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “While previous works also cast SKG tasks into the text-to-text format (Hosseini-Asl et al., 2020; Shaw et al., 2021; Liu et al., 2021), their independent choices of pretrained language models (PLMs), input-output formats, and frameworks make our uniﬁcation non-trivial.”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, methodology): “Following Shaw et al. (2021); Hazoom et al. (2021), we use T5 as our baseline model.”
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, methodology): “With the recent advances in pre-trained language models (PLMs), many existing works formulate the Text-to-SQL task as a semantic parsing problem and use a sequence-to-sequence (seq2seq) model to solve it (Scholak, Schucher, and Bahdanau 2021; Shi et al. 2021; Shaw et al. 2021).”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Our approach is closely related to NQG-T5 [98] which uses a rules-based approach to induce a non-probabilistic QCFG and then backs off to a ﬂex-ible sequence-to-sequence model during prediction if the grammar cannot parse the input sequence.”
Evaluating the Text-to-SQL Capabilities of Large Language Models (2022, methodology): “We compare GPT-3 and Codex against methods from Shaw et al. (2021) using the T5 encoder-decoder 1See Appendix A.2 for a discussion on parameter counts. ar X iv :2 20 4.”
Binding Language Models in Symbolic Languages (2022, methodology): “…grounding tasks mainly adopt semantic parsing since symbolic languages like SQL, SPARQL can be executed on them (Berant et al., 2013; Liang et al., 2017; Yin & Neubig, 2017; Zhong et al., 2018; Yu et al., 2018; Shaw et al., 2021; Scholak et al., 2021 (Rajkumar et al., 2022; Shi et al., 2022).”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “, 2020) have enjoyed dramatic success on semantic parsing (Lin et al., 2018; Hwang et al., 2019; Shaw et al., 2020; Suhr et al., 2020).”
Compositional Semantic Parsing with Large Language Models (2022, background): “However, SCAN is an artificial task built upon a synthetic language with a tiny vocabulary and is generated from a small set of grammar rules, and it is unclear whether strong results transfer to more realistic tasks that are based on a larger vocabulary and more complicated grammars (Furrer et al.,”
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models (2021, methodology): “We use the same serialization scheme used by Shaw et al. (2021).”
Span-based Semantic Parsing for Compositional Generalization (2020, background): “Finally, concurrently to us, Shaw et al. (2020) induced a synchronous grammar over program and utterance pairs and used it to introduce a compositional bias, showing certain improvements over compositional splits.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, background): “, 2020), language modeling (Andreas, 2020; Shaw et al., 2020), and text generation (Feng et al.”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “Full ﬁne-tuning of model parameters is a standard approach for applying LMs to end tasks, and T5 performance with ﬁne-tuning has been measured for compositional generalization up to the scale of 11 billion parameters (Shaw et al., 2021; Furrer et al., 2020).”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, background): “For both our T5 baseline and LIR d +RIR, further increasing model capacity beyond T5-base does not give further improvements, which is consistent with previous work on similar tasks with small train set sizes (Shaw et al., 2020; Furrer et al., 2020).”