Improving Text-to-SQL Evaluation Methodology

By Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir R. Radev. Annual Meeting of the Association for Computational Linguistics, 2018

311 citations recorded by the source corpus.

To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/2f541b24f69798e255e04229e8dc78f4d1873fd7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Natural language to SQL: Where are we today? (2020, methodology): “For the Advising benchmark, we use two versions of splits published in [13], namely question-based split and query-based split.”
Robust Text-to-SQL Generation with Execution-Guided Decoding (2018, methodology): “Table 2 shows the results for the template-based model (Finegan-Dollak et al. 2018) and the sequence-tosequence model (Iyer et al.”
Sequence-level Mixed Sample Data Augmentation (2020, methodology): “Semantic Parsing For semantic parsing, we consider the SQL queries subset of GeoQuery (Finegan-Dollak et al., 2018), which consists of 880 English questions paired with SQL commands.”
Evaluating the Text-to-SQL Capabilities of Large Language Models (2022, methodology): “Few-Shot Experiments We re-purpose the question-splits of the GeoQuery and Scholar datasets (Zelle and Mooney, 1996; Iyer et al., 2017; Finegan-Dollak et al., 2018) to perform experiments in a few-shot setting.”
Grammar-based Neural Text-to-SQL Generation (2019, background): “There has recently been work in standardizing the many proposed text-to-SQL datasets (Finegan-Dollak et al., 2018).”
Learning Programmatic Idioms for Scalable Semantic Parsing (2019, background): “Finegan-Dollak et al. (2018) observed that the SQL structures in ATIS-SQL are repeated numerous times in both train and test sets, thus facilitating Seq2seq models to memorize these structures without explicit idiom supervision.”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, methodology): “For prefix-to-SQL, we build the PAGSAS benchmark, which consists of 5 sub-tasks and 124K examples from Advising (Finegan-Dollak et al., 2018), GeoQuery (Zelle and Mooney, 1996a), Scholar (Iyer et al.”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “(2021): (1) the template split, where abstract output templates in training and test data are disjoint (Finegan-Dollak et al., 2018); (2) the TMCD split, which makes the distributions of compounds in training and test data as divergent as possible; and (3) the length split.”
Benchmarking Meaning Representations in Neural Semantic Parsing (2020, background): “study (Finegan-Dollak et al., 2018) in Text-to-SQL has shown that domain-specific MRs are more ro-”
SParC: Cross-Domain Semantic Parsing in Context (2019, methodology): “The problem of mapping a natural language utterance into executable SQL queries (text-to-SQL) has attracted increasing attention from the semantic parsing community by virtue of a continuous effort of dataset creation (Zelle and Mooney, 1996; Iyyer et al., 2017; Zhong et al., 2017; Finegan-Dollak et”
Few-Shot Semantic Parsing for New Predicates (2021, methodology): “Moreover, the SOTA parsers achieve less than 32% of accuracy on ﬁve widely used corpora, when the LFs in the test sets do not share LF templates in the training sets (Finegan-Dollak et al., 2018).”
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (2019, background): “, 2018c) have received an increasing amount of attention as many data-driven neural approaches achieve promising results (Dong and Lapata, 2016; Su and Yan, 2017; Iyer et al., 2017; Xu et al., 2017; Finegan-Dollak et al., 2018; Yu et al., 2018a; Huang et al., 2018; Dong and Lapata, 2018; Sun et al.,”
Photon: A Robust Cross-Domain Text-to-SQL System (2020, methodology): “Spider (Finegan-Dollak et al., 2018) specifically controlled question clarify during data collection to exclude poorly phrased and ambiguous questions.”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, background): “While most work focus on single tables, often obtained from the Web, some have extended modeling to more complex structures such as relational databases (Finegan-Dollak et al., 2018; Yu et al., 2018b; Wang et al., 2020).”