Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

By Tao Yu, Rui Zhang, Kai-Chou Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Z Li, Qingning Yao, Shanelle Roman and 2 more. 2018

1,963 citations recorded by the source corpus.

We present Spider, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task so that different complicated SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Therefore, Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and have the exact same program in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 9.7% exact matching accuracy on a database split setting. This shows that Spider presents a strong challenge for future research. Our dataset and task with the most recent updates are publicly available at https://yale-lily.github.io/seq2sql/spider.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “Such domain shifts typically occur across related but different datasets (Torralba & Efros, 2011; Fang et al., 2013; Venkateswara et al., 2017; Yu et al., 2018b; 2019; Peng et al., 2019; Kamath et al., 2020), as well as from synthetic data to real data (including sim2real) (Wang et al., 2015; Ganin…”
SPoC: Search-based Pseudocode to Code (2019, background): “While semantic parsing datasets with shorter, query-like programs [57, 7, 33, 4, 59, 23, 44, 55] usually have an execution environment (e.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “…tasks formatted as text-to-text: DART (Nan et al., 2020), which is a held-out datato-text task for transforming data triples to text, Spider (Yu et al., 2018), a SQL query generation task given a database and an input query, and fully supervised in our framework, and MultiWoZ (Budzianowski…”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
Break It Down: A Question Understanding Benchmark (2020, methodology): “…Collection Questions in BREAK were randomly sampled from ten QA datasets over the following tasks (Table 3): • Semantic Parsing: Mapping natural language utterances into formal queries, to be executed on a target KB (Price, 1990; Zelle and Mooney, 1996; Li and Jagadish, 2014; Yu et al., 2018).”
Learning by Distilling Context (2022, methodology): “Section 3.2 (Figure 4) shows that context distillation can be used to internalize Text-to-SQL training examples from the SPIDER dataset (Yu et al., 2018) into Incoder (Fried et al., 2022), and it outperforms directly learning with gradient descent by 9% for 8-shot adaptation; additionally, we show…”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “Fol1340 lowing Nan et al. (2021b), each relation-triple 1341 is linearized as “sub : rela : obj”, and different 1342 triples are joined by “ | ”.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “We do not include two previously studied languages, C [6] and SQL [27], [28] because they do not meet this criterion.”
SantaCoder: don't reach for the stars! (2023, background): “Many single-language benchmarks for evaluating code completion exist (Kulal et al., 2019; Iyer et al., 2018; Zhong et al., 2017; Yu et al., 2018; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Athiwaratkun et al., 2022; Lai et al., 2022).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…et al. 2017 sick cls/nli Marelli et al. 2014 sms_spam cls/other Almeida et al. 2011 social_i_qa qa/multiple-choice qa Sap et al. 2019 spider cg/other Yu et al. 2018 squad-no_context qa/closed-book qa Rajpurkar et al. 2016 squad-with_context qa/machine reading comprehension Rajpurkar et al. 2016…”
Measuring Coding Challenge Competence With APPS (2021, methodology): “We compared to several datasets in the (Kulal et al., 2019; Yu et al., 2018; Raychev et al., 2016; Iyer et al., 2018; Lu et al., 2021) main paper.”
TabFact: A Large-scale Dataset for Table-based Fact Verification (2019, neutral): “, 2014; Artzi & Zettlemoyer, 2013), code generation (Yin & Neubig, 2017; Dong & Lapata, 2016), SQL synthesis (Yu et al., 2018), etc.”
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020, background): “…of the XXXI Olympiad (Portuguese : Jogos da XXXI Olimpíada) and commonly known as Rio 2016 , was an international multi-sport event …… Be-rant et al., 2013), WebQSP (Yih et al., 2016), Web-Complex (Talmor and Berant, 2018), MetaQA (Zhang et al., 2018), WikiTableQuestion (Pasupat and Liang, 2015).”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “There has been work on translating natural language to code in different languages (Zettlemoyer and Collins, 2012; Wang et al., 2015; Oda et al., 2015; Yin et al., 2018; Zhong et al., 2017; Yu et al., 2018; Lin et al., 2018).”
Towards Ecologically Valid Research on Language User Interfaces (2020, neutral): “In the context of database QA, Spider (Yu et al., 2018) collected questions from 11 computer science students with proﬁciency in SQL.”