Semantic Evaluation for Text-to-SQL with Distilled Test Suite

By Ruiqi Zhong, Tao Yu, Daniel Klein. 2020

190 citations recorded by the source corpus.

We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models. Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases. At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently. We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples. In contrast, the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed. Our implementation, along with distilled test suites for eleven Text-to-SQL datasets, is publicly available.

Subfields: Artificial Intelligence, Information Systems and Management, Management Science and Operations Research, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Management Science and Operations Research

In Code Generation & Program Synthesis

Citations in the corpus

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “For most semantic parsing tasks, we report the exact match accuracy of logical forms, and for task has test suite (Zhong et al., 2020), we add test suite metric to represent model’s performance; an exception is WebQSP, for which we follow previous work to execute the parses and report the F1 score.”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “These metrics diverge from the programmer’s intent, increasingly so as model capability improves (Zhong et al., 2020).”
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, methodology): “Evaluation Metrics To evaluate the performance of the Text-to-SQL parser, following Yu et al. 2018c; Zhong, Yu, and Klein 2020, we adopt two metrics: Exact-set-Match accuracy (EM) and EXecution accuracy (EX).”
SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (2024, methodology): “We consider two commonly-used evaluation metrics: execution accuracy (EX) and test-suite accuracy (TS) [32].”
LLMs Get Lost In Multi-Turn Conversation (2025, unknown): “Zhong et al. [90] found that by evaluating on an increased number of databases, false positives become negligible.”
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, methodology): “We manually examined these failures and classiﬁed them into six categories as shown in Figure 1 and discussed next.”
Natural Language to Code Translation with Execution (2022, background): “While all the above evaluations are based on executing one test case per problem, more test cases can lead to more accurate judgments of semantic equivalence between programs (Zhong et al., 2020).”
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models (2021, methodology): “This metric is sensitive to literal query values, but suffers from a high false positive rate (Zhong et al., 2020).”
Evaluating the Text-to-SQL Capabilities of Large Language Models (2022, methodology): “The latter metric was proposed by Zhong et al. (2020) to measure semantic equivalence of SQL queries written in different styles, which is essential when comparing Codex to models trained on Spider.”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, methodology): “3 We evaluate the methods by accuracy for all classiﬁcation and multiple-choice selection datasets, joint accuracy (Budzianowski et al., 2018) for MWoZ, test suite accuracy (Zhong et al., 2020) for GeoQuery, exact matching (Rajpurkar et al., 2016) for NQ, and ROUGE-L (Lin, 2004) for XSum.”
CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, unknown): “The EX metric evaluates whether the predicted and ground-truth SQL queries yield the same execution results on the database.”
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, methodology): “To make a fair comparison, we follow prior study [51] to use exact-set-match accuracy (EM) and execution accuracy (EX).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, methodology): “However, CONCODE (Iyer et al., 2018) is the only language-to-code task included in CodeXGLUE and it uses surface-form based evaluation metrics such as BLEU.”