Evaluating the Text-to-SQL Capabilities of Large Language Models

By Nitarshan Rajkumar, Raymond Li, Dzmitry Bahdanau. 2022

158 citations recorded by the source corpus.

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning by Distilling Context (2022, methodology): “We choose this task because prior work (Rajkumar et al., 2022) has demonstrated that SOTA models are able to effectively utilize in-context examples for this task, and we conjecture that our approach is more likely to succeed since this task requires complex cognitive reasoning.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “…1 9 D ec 2 02 2 cutable programs (Chen et al., 2021a; Austin et al., 2021; Chowdhery et al., 2022; Nijkamp et al., 2022; Fried et al., 2022, inter alia), with promising applications in synthesizing code for data wrangling and EDA tasks (Jain et al., 2021; Rajkumar et al., 2022; Cheng et al., 2022).”
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023, methodology): “We employ programming-based prompts, as described in [Rajkumar et al., 2022], to collect results by calling the API.”
LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, methodology): “Previous Work without Finetuning Rajkumar et al. (2022) 67.0 MBR-Exec (Shi et al., 2022) 75.2 Coder-Reviewer (Zhang et al., 2022) 74.5 Previous Work with Finetuning T5-3B (Xie et al., 2022) 71.8 PICARD (Scholak et al., 2021) 75.5 RASAT (Qi et al., 2022) 80.5 This Work with code-davinci-002 Greedy 75”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, background): “, 2023), has shown that such LLMs that are trained on a mixture of text and code are able to perform language-to-code generation under few-shot or even zero-shot learning settings (Rajkumar et al., 2022; Ni et al., 2023).”
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, methodology): “For instance, the database schema for question “What are the average and maximum capacities for all stadiums?” included a column named “aver-age”, which was selected by the model instead of taking the average of the capacity column.”
Teaching Large Language Models to Self-Debug (2023, methodology): “Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, background): “In recent years, large language models (LLMs) have emerged as a new paradigm for Text-to-SQL [22, 34, 41].”
Binding Language Models in Symbolic Languages (2022, methodology): “We base our experiments for B INDER on Codex, a code-pretrained version of GPT-3, which has been shown to perform proﬁciently on code generation tasks (Rajkumar et al., 2022; Chen et al., 2022) and even some weakly-related tasks like dialogue state tracking (Hu et al., 2022).”
SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (2024, methodology): “GPT3 and the rest of the CodeX results are from [31].”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “with in-context learning using GPT-3 (Brown et al., 2020; Shin et al., 2021; Shin and Durme, 2021; Rubin et al., 2021; Rajkumar et al., 2022), they do not focus on compositional generalization.”