Natural language to SQL: Where are we today?

By Byeong-Hoon So, Wook-Shin Han, Hongrae Lee. 2020

156 citations recorded by the source corpus.

Translating natural language to SQL (NL2SQL) has received extensive attention lately, especially with the recent success of deep learning technologies. However, despite the large number of studies, we do not have a thorough understanding of how good existing techniques really are and how much is applicable to real-world situations. A key difficulty is that different studies are based on different datasets, which often have their own limitations and assumptions that are implicitly hidden in the context or datasets. Moreover, a couple of evaluation metrics are commonly employed but they are rather simplistic and do not properly depict the accuracy of results, as will be shown in our experiments. To provide a holistic view of NL2SQL technologies and access current advancements, we perform extensive experiments under our unified framework using eleven of recent techniques over 10+ benchmarks including a new benchmark (WTQ) and TPC-H. We provide a comprehensive survey of recent NL2SQL methods, introducing a taxonomy of them. We reveal major assumptions of the methods and classify translation errors through extensive experiments. We also provide a practical tool for validation by using existing, mature database technologies such as query rewrite and database testing. We then suggest future research directions so that the translation can be used in practice.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “, 2020), database commands (Kim et al., 2020), programming instructions (Lin et al.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “…are a versatile way of deﬁn-ing goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shrid-har et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database commands (Kim et al., 2020).”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “…are a versatile way of deﬁning goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shrid-har et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database commands (Kim et al., 2020).”
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (2021, background): “For example, mapping natural language to database commands (Kim et al., 2020), visualizations (Shao and Nakashole, 2020), console commands (Lin et al.”