Constrained Language Models Yield Few-Shot Semantic Parsers

By Richard Shin, C. H. Lin, Sam Thomson, Charles C. Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Daniel Klein, Jason Eisner, Benjamin Van Durme. 2021

247 citations recorded by the source corpus.

We explore the use of large pretrained language models as few-shot semantic parsers. The goal in semantic parsing is to generate a structured meaning representation given a natural language input. However, language models are trained to generate natural language. To bridge the gap, we use language models to paraphrase inputs into a controlled sublanguage resembling English that can be automatically mapped to a target meaning representation. Our results demonstrate that with only a small amount of data and very little code to convert into English-like representations, our blueprint for rapidly bootstrapping semantic parsers leads to surprisingly effective performance on multiple community tasks, greatly exceeding baseline methods also trained on the same limited data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “[124] explore the task of few-shot semantic parsing using LMs by (1) framing the semantic parsing task as a paraphrasing task [7] and (2) constraining the decoding process by only allowing output valid according to a grammar.”
PAL: Program-aided Language Models (2022, methodology): “Some works constrain the decoder using a Context-Free Grammar (CFG) to generate a domain-specific meaning representation (Shin & Van Durme, 2021) or a canonical utterance, which can be converted to a Lisp-like meaning representation (Shin et al., 2021).”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “We found that this simple parsing technique works well and could easily be applied to other parsing-based tasks, as in Shin et al. (2021).”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, background): “Choosing closer data points for prompting is a common and effective way of scaling up the training data size for in-context learning (Shin et al., 2021; Liu et al., 2021).”
Compositional Semantic Parsing with Large Language Models (2022, methodology): “We use Codex instead of text-based GPT-3 in our evaluation because of its better performance in our initial experiments, and this observation is also presented in prior works on semantic parsing (Shin et al., 2021; Shin & Van Durme, 2022).”
Few-Shot Semantic Parsing with Language Models Trained on Code (2021, methodology): “We revisit few-shot semantic parsing experiments from Shin et al. (2021), which used GPT-3 with constrained decoding into a controlled sublan-guage of English (canonical utterances) then translated the canonical utterance output into the meaning representation using a synchronous context-free…”
SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022, methodology): “One is tuning-free prompting, for example, Petroni et al. (2019); Shin et al. (2020) used a ﬁll-in-the-blank paradigm, while Brown et al. (2020); Shin et al. (2021) used “few-shot” prompts that included several examples of inputs followed by target outputs, with the actual task input appended at…”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “, 2022), use tools and APIs (Shin et al., 2021; Thoppilan et al., 2022), and answer open-domain questions (Petroni et al.”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “, 2022), use tools and APIs (Shin et al., 2021; Thoppilan et al., 2022), and answer open-domain questions (Petroni et al.”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, background): “Choosing closer data points for prompting is a common and effective way of scaling up the training data size for in-context learning (Shin et al., 2021; Liu et al., 2021).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “For instance, semantic parsing, generating a structure representing the semantics of the sentence, is explored by Shin et al. [169].”
Synchromesh: Reliable code generation from pre-trained language models (2022, methodology): “In many cases, this methodology was shown to provide good performance, reducing the need to annotate large datasets for each task of interest (Brown et al., 2020; Shin et al., 2021).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “Prior work has shown the potential of leveraging alternative output representations to improve semantic parsers (Herzig et al., 2021; Shin et al., 2021).”