Few-Shot Semantic Parsing with Language Models Trained on Code

By Richard Shin, Benjamin Van Durme. 2021

74 citations recorded by the source corpus.

Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pre-trained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

PAL: Program-aided Language Models (2022, methodology): “Some works constrain the decoder using a Context-Free Grammar (CFG) to generate a domain-specific meaning representation (Shin & Van Durme, 2021) or a canonical utterance, which can be converted to a Lisp-like meaning representation (Shin et al., 2021).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “with in-context learning using GPT-3 (Brown et al., 2020; Shin et al., 2021; Shin and Durme, 2021; Rubin et al., 2021; Rajkumar et al., 2022), they do not focus on compositional generalization.”