CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
By Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Y. Tan, Tianze Shi, Zihan Li and 14 more. 2019
312 citations recorded by the source corpus.
We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets: (1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.
Subfields: Artificial Intelligence, Dialogue & Conversational AI, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Caiming Xiong
- Jianfeng Gao
- Richard Socher
- Noah A. Smith
In Dialogue & Conversational AI
In Code Generation & Program Synthesis
- Graham Neubig
- Frank F. Xu
- Lizhen Qu
- Gholamreza Haffari
- Zhuang Li
- Bogdan Vasilescu
- Luke S. Zettlemoyer
- Pengcheng Yin
Citations in the corpus
- PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “…Collins, 2012; Oda et al., 2015; Wang et al., 2015; Yin et al., 2018), and with addition information such as database schemas (Zhong et al., 2017; Yu et al., 2018, 2019b,a) or input-output examples (Polosukhin and Skidanov, 2018; Zavershynskyi et al., 2018), synthesizing general-purpose code…”
- Logic-Consistency Text Generation from Semantic Parses (2021, background): “…and usability of the latest natural language interfaces (Gatt and Krahmer, 2018; Chen et al., 2020b; Hu et al., 2020; Mishra et al., 2019; Yu et al., 2019; Ngomo et al., 2013; Wang et al., 2018; Gardent et al., 2017; Wang et al., 2020a; *Equal Contribution Figure 1: Our data augmentation…”
- Investigating Pretrained Language Models for Graph-to-Text Generation (2020, background): “Recently, it has been shown that structured meaning representation, such as AMR or KG, can store the internal state of a dialog sys-tem, providing core semantic knowledge (Bonial et al., 2020; Bai et al., 2021) or can be the result of a database query for conversational QA (Yu et al., 2019).”
- Semantic Evaluation for Text-to-SQL with Distilled Test Suite (2020, methodology): “We distill one shared test suite for the development set of S pider (Yu et al., 2018), C o SQL (Yu et al., 2019a) and SP ar C (Yu et al., 2019b), since they share the same 20 database schemata.”
- Context Dependent Semantic Parsing: A Survey (2020, methodology): “Different from (Yu et al., 2019a), Vlachos and Clark (2014) integrated the dialogue acts into the MRs .”
- Teaching Large Language Models to Self-Debug (2023, background): “, 2023a), interact with users for multi-turn code generation (Yu et al., 2019; Yao et al., 2019; Nijkamp et al., 2023), and fix generated programs based on human feedback (Iyer et al.”
- Grounded Adaptation for Zero-shot Executable Semantic Parsing (2020, methodology): “Second, we divide multi-turn queries into how many turns into the interaction they occur for Sparc and CoSQL (Yu et al., 2019b,a).”
- Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “The majority of this work has focused on mapping a single query to the corresponding SQL with the exception of a few datasets, e.g., SParC (Yu et al., 2019b) and CoSQL (Yu et al., 2019a), that target inducing SQL parses for sequentially related questions.”
- Learning Adaptive Language Interfaces through Decomposition (2020, background): “More recently, Yu et al. (2019) introduced the large-scale Conversational Text-to-SQL (CoSQL) dataset that requires jointly reasoning over dialogue history and databases to parse user queries to SQL.”
- NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, background): “They contrast the task with conversational semantic parsing (Suhr et al., 2018; Yu et al., 2019b,a; Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context.”
- On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020, unknown): “The family of Spider datasets (Yu et al., 2018, 2019a,b) contain queries even more complex than in WTQ, including a higher percentage of nested queries and multiple table joins.”
- ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “…Sha et al., 2018; Liu et al., 2018; Perez-Beltrachini and Lapata, 2018), descriptions of knowledge bases (ODonnell et al., 2000; Trisedya et al., 2018; Zhu et al., 2019; Yu et al., 2019) and source code (Iyer et al., 2016), and dialog response generation from slot-value pairs (Wen et al., 2015).”
- Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “In CoSQL (Yu et al., 2019a), users engage in a dialogue with a conversational database interface that is enacted by a SQL-competent wiz-ard.”
- Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (2019, methodology): “We compare with the two baseline models released by Yu et al. (2019b).”