DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction

By M. Pourreza, Davood Rafiei. Neural Information Processing Systems, 2023

761 citations recorded by the source corpus.

We study the problem of decomposing a complex text-to-sql task into smaller sub-tasks and how such a decomposition can significantly improve the performance of Large Language Models (LLMs) in the reasoning process. There is currently a significant gap between the performance of fine-tuned models and prompting approaches using LLMs on challenging text-to-sql datasets such as Spider. We show that SQL queries, despite their declarative structure, can be broken down into sub-problems and the solutions of those sub-problems can be fed into LLMs to significantly improve their performance. Our experiments with three LLMs show that this approach consistently improves their performance by roughly 10%, pushing the accuracy of LLMs towards state-of-the-art, and even beating large fine-tuned models on the holdout Spider dataset.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/cc0f0cb09a73f82ed44d900f5ca710bec784acc1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, neutral): “( L3 ) Most closed-source models have numerous parameters (e.g., 175B parameters for GPT-3.”
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023, background): “The latest SOTA parser [Pourreza and Rafiei, 2023] in Spider benefits from the powerful understanding and coding capabilities of the large language model (LLM), and such excellent performance leads us to ask a question: Can LLM already serve as a database interface ? The answer is no, as shown in Fi”
SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (2024, methodology): “They have dominated the state-of-the-art for a long time, and have been only very recently outperformed by one in-context learning approach [2].”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “…but interest has been growing in building multi-stage pipelines and agents that decompose complex tasks into more manageable calls to LMs in an effort to improve performance (Qi et al., 2019; Khattab et al., 2021a; Karpas et al., 2022; Dohan et al., 2022; Pourreza & Rafiei, 2023; Yao et al., 2023).”
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, methodology): “For token cost, we average the token number of 10 randomly sampled instances for DIN-SQL.”
Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval (2026, unknown): “Schema linking, identifying the schema elements that a query requires, is a well-studied sub-problem (Li et al., 2023a; Pourreza and Rafiei, 2023; Gao et al., 2024); as schemas grow beyond what fits in context, it is increasingly implemented as a retrieval step whose output is passed to an LLM,…”