Compositional Semantic Parsing with Large Language Models

By Andrew Drozdov, Nathanael Scharli, Ekin Akyuurek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, Denny Zhou. 2022

113 citations recorded by the source corpus.

Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable large language models (LLMs) to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “, 2019), or solve the sub-problems sequentially, where the solution to the next sub-problem depends on the answer to the previous ones (Yang et al., 2022a; Zhou et al., 2022; Drozdov et al., 2022; Dua et al., 2022; Khot et al., 2022; Wang et al., 2022a; Wu et al., 2022b; Mishra and Nouri, 2022).”
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022, background): “…reasoning steps) can substantially improve the performance of language models on a range of tasks involving arithmetic reasoning, common-sense reasoning, symbolic manipulation, and program execution (Wei et al., 2022b; Zhou et al., 2022; Drozdov et al., 2022; Nye et al., 2021, inter alia ).”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “Later on, several other works (Drozdov et al., 2022; Zhou et al., 2022; Nye et al., 2021) also propose different approaches to utilize LLMs to solve reasoning tasks by allowing intermediate steps.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “, reasoning (Wei et al., 2022b; Wang et al., 2022; Zhou et al., 2022; Drozdov et al., 2022), math (Lewkowycz et al.”
Large Language Models as Analogical Reasoners (2023, background): “Several studies have extended this approach with more structured algorithms and search methods (Khot et al., 2022; Drozdov et al., 2022; Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as longer-horizon action and planning (Yao et al.,…”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “…et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery…”
Large Language Models are few(1)-shot Table Reasoners (2022, background): “Wei et al. (2022); Wang et al. (2022); Zhou et al. (2022); Drozdov et al. (2022) have recently discovered that large language models (Brown et al., 2020; Chowdhery et al., 2022; Ouyang et al., 2022) can be used to solve complex mathematical and commonsense reasoning tasks with few-shot in-context…”
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (2024, background): “…Chung et al., 2022; Huang et al., 2022), dramatically enhances the performance of large language models (LLMs) on various reasoning benchmarks (Wang et al., 2023; Lewkowycz et al., 2022; Suzgun et al., 2022; Shi et al., 2023; Yao et al., 2022; Drozdov et al., 2022; OpenAI, 2023; Google, 2023).”
Is GPT-3 a Good Data Annotator? (2022, background): “In instances where more intricate tasks are encountered, previous research endeavors (Drozdov et al., 2023; Liu et al., 2023; Qin et al., 2023b) have demonstrated the remarkable capabilities of LLMs in complex NLP tasks, encompassing reasoning, semantic parsing, and dialogues, even in situations…”
Towards Reasoning in Large Language Models: A Survey (2022, background): “As follow-up work, Drozdov et al. (2022) introduce dynamic least-to-most prompting, which is designed to solve more realistic semantic parsing problems by decomposing the problems with prompting-based syntactic parsing”
Can large language models reason about medical questions? (2022, background): “CoT prompting led to substantial improvements on many reasoning-intensive tasks (Wei et al., 2022; Zhou et al., 2022; Drozdov et al., 2022; Nye et al., 2021), allowing to bridge the gap with human-level performances for most of the hard BIG-bench tasks (Suzgun et al.”
Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “One possible solution is to use a chain-ofthought approach to break down complex problems (Zhou et al. 2023a; Xu et al. 2023b; Drozdov et al. 2023).”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, unknown): “Later on, several other works (Drozdov et al., 2023; Zhou et al., 2023c; Nye et al., 2022; Wang et al., 2022a; 2023a; Li et al., 2023b; Wang et al., 2023d; Yu et al., 2023) (Chen et al., 2022).”