CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

By Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Y. Tan, Tianze Shi, Zihan Li and 14 more. 2019

312 citations recorded by the source corpus.

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets: (1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

Subfields: Artificial Intelligence, Dialogue & Conversational AI, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Graham Neubig
  2. Mohit Bansal
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Caiming Xiong
  6. Jianfeng Gao
  7. Richard Socher
  8. Noah A. Smith

In Dialogue & Conversational AI

  1. Caiming Xiong

In Code Generation & Program Synthesis

  1. Graham Neubig
  2. Frank F. Xu
  3. Lizhen Qu
  4. Gholamreza Haffari
  5. Zhuang Li
  6. Bogdan Vasilescu
  7. Luke S. Zettlemoyer
  8. Pengcheng Yin

Citations in the corpus