A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

By Wonseok Hwang, Jin A Yim, Seunghyun Park, Minjoon Seo. 2019

276 citations recorded by the source corpus.

We present SQLova, the first Natural-language-to-SQL (NL2SQL) model to achieve human performance in WikiSQL dataset. We revisit and discuss diverse popular methods in NL2SQL literature, take a full advantage of BERT {Devlin et al., 2018) through an effective table contextualization method, and coherently combine them, outperforming the previous state of the art by 8.2% and 2.5% in logical form and execution accuracy, respectively. We particularly note that BERT with a seq2seq decoder leads to a poor performance in the task, indicating the importance of a careful design when using such large pretrained models. We also provide a comprehensive analysis on the dataset and our model, which can be helpful for designing future NL2SQL datsets and models. We especially show that our model's performance is near the upper bound in WikiSQL, where we observe that a large portion of the evaluation errors are due to wrong annotations, and our model is already exceeding human performance by 1.3% in execution accuracy.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Natural Language Processing

Citations in the corpus

A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “…et al., 2019) 74.9 74.8 MML 70.6 70.5 Ours 84.4 83.9 Fully-supervised setting SQLNet (Xu et al., 2018) 69.8 68.0 TypeSQL (Yu et al., 2018b) 74.5 73.5 Coarse2Fine (Dong and Lapata, 2018) 79.0 78.5 SQLova (Hwang et al., 2019) 87.2 86.2 X-SQL (He et al., 2019) 89.5 88.7 Table 4: Results on W IKI SQL.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “To encode structured knowledge, prior work linearized the structured knowledge and concatenated it with the text (Hwang et al., 2019; Liu et al., 2020; Hosseini-Asl et al., 2020; Liu et al., 2021), which has been augmented by positional encoding (e.g., row/column embedding) (Herzig et al., 2020;…”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “This observation has been successfully utilized in previous works (Toutanova and Chen, 2015; Hwang et al., 2019).”
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020, methodology): “Existing works have applied different linearization strategies to encode tables with Transformers (Hwang et al., 2019; Chen et al., 2019), while our row approach is specifically designed for encoding content snapshots.”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, methodology): “As shown in the bottom two lines of Table 4, G RAPPA improves the performance of the SQLova model by 3.0% compared to RoBERTa, indicating that G RAPPA can make the base parser more sample-efﬁcient.”
Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing (2021, methodology): “Same as in (Hwang et al. 2019), we concatenate all utterances in one conversation and all the schema-related input using [SEP] as the delimiter:”
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (2019, methodology): “To be speciﬁc, we follow Hwang et al. (2019) to concatenate the user utterance and all the column headers in a single sequence separated by the [SEP] token: This sequence is fed into the pretrained BERT model whose hidden states at the last layer is used as the input embedding.”
TaPas: Weakly Supervised Table Parsing via Pre-training (2020, background): “Additional embeddings We add a separator token between the question and the table, but unlike Hwang et al. (2019) not between cells or rows.”
Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation (2019, methodology): “…manner and achieve beyond 80% exact matching accuracy on public Text-to-SQL benchmarks (e.g., ATIS, GeoQuery and WikiSQL) (Krishna-murthy et al., 2017; Zhong et al., 2017; Xu et al., 2017; Yaghmazadeh et al., 2017; Yu et al., 2018a; Dong and Lapata, 2018; Wang et al., 2018; Hwang et al., 2019).”
Learning to Solve Complex Tasks by Talking to Agents (2021, background): “This is meant to emulate how black-box models are generally trained on datasets with table inputs (Herzig et al., 2020; Hwang et al., 2019).”
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, methodology): “Sequence-to-sequence models (Sutskever et al., 2014) have shown great potential in code generation tasks including text-to-SQL The key idea is to jointly encode a given natural language question together with the database schema and leverage a decoder to predict the target SQL On the encoder side, l”
Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (2020, methodology): “Contextual embeddings like BERT are later shown to be successful for cross-domain text-to-SQL parsing [19].”
Photon: A Robust Cross-Domain Text-to-SQL System (2020, methodology): “Following previous work (Hwang et al., 2019; Zhang et al., 2019), we serialize the relational DB schema and concatenate it to the user question.”
X-SQL: reinforce schema representation with context (2019, methodology): “On the other hand, for dev set human performance is estimated to be 88.2% according to Hwang et al. (2019).”
Improving Entity Linking by Modeling Latent Entity Type Information (2020, result): “Similar learning rate usage can be found in the recent work by (Hwang et al. 2019).”
Semantic Evaluation for Text-to-SQL with Distilled Test Suite (2020, methodology): “…queries from the following eight models: MQAN unordered (McCann et al., 2018), X-SQL (He et al., 2019), HydraNet with / without Execution Guidance (Lyu et al., 2020), IncSQL (Shi et al., 2018), SQLova with / without Execution Guidance (Hwang et al., 2019) and HardEM (Min et al., 2019).”
ReTraCk: A Flexible and Efficient Framework for Knowledge Base Question Answering (2021, methodology): “BERT Encoding Motivated by the success of pretrained language models on cross-domain text-to-SQL tasks (Hwang et al., 2019), we augment our model with BERT (Devlin et al., 2019).”
Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study (2019, methodology): “We experiment MISP-SQL with two base semantic parsers: SQLNet (Xu et al., 2017) and SQLova (Hwang et al., 2019).”