Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

By Victor Zhong, Caiming Xiong, Richard Socher. 2017

1,572 citations recorded by the source corpus.

Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Reinforcement Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Reinforcement Learning

In Signal Processing

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “On WikiSQL, it obtains a much higher score than prior sequence-to-sequence baselines [Zhong et al., 2017], but it is low compared to MQAN (+QPtr) and the other baselines.”
Reasoning Like Program Executors (2022, background): “, 2020) over databases provided by WIKISQL (Zhong et al., 2017), 5 million examples are synthesized for pre-training.”
SPoC: Search-based Pseudocode to Code (2019, background): “Previous work on executable semantic parsing mainly focuses on translating short text descriptions to a one-line program [42, 35, 43, 44, 21, 8], and while recent work explored generating longer programs from text descriptions [22, 40, 29, 15, 16, 14], these programs are mostly evaluated on…”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “This area has attracted growing interest, with applications such as converting natural language to Excel macros [77], Java expressions [82], shell commands [118, 119], simple if-then programs [161], regular expressions [110], and to SQL queries [200].”
Augmented Language Models: a Survey (2023, background): “While there are early works focused on models that could interface with external tools, they employ ad-hoc tool-dependent architectures (Adolphs et al., 2022; Buck et al., 2018; Nogueira and Cho, 2017; Zhong et al., 2018).”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “Semantic Parsing W IKI SQL (Zhong et al., 2017) 56,355 8,421 15,878 346.1 5 Table 1: Six QA datasets in three different categories used in this paper (detailed in Section 5) along with the size of each dataset.”
UL2: Unifying Language Learning Paradigms (2022, background): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “…multiple complementary methods: (1) human annotation on open-domain Wikipedia tables from WikiTableQuestions (Pasupat and Liang, 2015) and WikiSQL (Zhong et al., 2017) (Section 3.1), (2) automatic conversion of questions in WikiSQL to declarative sentences (Section 3.2), and (3) incorporation of…”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “Question Answering WikiSQL (Zhong et al., 2017) Table Question Answer WikiTQ (Pasupat and Liang, 2015) Table Question Answer CompWebQ (Talmor and Berant, 2018) Knowledge Graph Question Answer HybridQA (Chen et al.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “Zhong et al. [28] and Yu et al. [38] propose benchmarks for SQL.”
SantaCoder: don't reach for the stars! (2023, background): “Many single-language benchmarks for evaluating code completion exist (Kulal et al., 2019; Iyer et al., 2018; Zhong et al., 2017; Yu et al., 2018; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Athiwaratkun et al., 2022; Lai et al., 2022).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “Execution-based Evaluation Evaluation by execution has long been used for SQL (Zhong et al., 2017) or logical forms (Dong and Lapata, 2016).”
The Web as a Knowledge-Base for Answering Complex Questions (2018, methodology): “Because we also need to decode the tokens C OMP and C ONJ , we use “augmented pointer networks”, (Zhong et al., 2017): For every question q , an augmented question ˆ q is created by appending the to-kens “C OMP C ONJ ” to q .”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…Jiang et al. 2020 wiki_bio cg/other Lebret et al. 2016 wiki_qa cls/other Yang et al. 2015 wiki_split cg/other Botha et al. 2018 wikisql cg/other Zhong et al. 2017 wino_grande qa/multiple-choice qa Sakaguchi et al. 2020 wiqa qa/multiple-choice qa Tandon et al. 2019 xsum cg/summarization…”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, background): “Second, existing datasets that are large in terms of the number of programs and databases such as WikiSQL (Zhong et al., 2017) contain only simple SQL queries and single tables.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “In this case, we further incorporate Wik-iSQL (SQL query generation, Zhong et al., 2017), CNN/DailyMail (news article summarization See et al., 2017), MultiWOZ (semantic state sequence generation (Budzianowski et al., 2018)) into our task sequences 1 .”
LAMOL: LAnguage MOdeling for Lifelong Language Learning (2019, neutral): “• Semantic Parsing – WikiSQL (Zhong et al., 2017) • News Classiﬁcation – AGNews: News articles to be classiﬁed into 4 classes.”