SParC: Cross-Domain Semantic Parsing in Context

By Tao Yu, Rui Zhang, Michihiro Yasunaga, Y. Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Z Li, Bo Pang, Tao Chen and 9 more. 2019

251 citations recorded by the source corpus.

We present SParC, a dataset for cross-domainSemanticParsing inContext that consists of 4,298 coherent question sequences (12k+ individual questions annotated with SQL queries). It is obtained from controlled user interactions with 200 complex databases over 138 domains. We provide an in-depth analysis of SParC and show that it introduces new challenges compared to existing datasets. SParC demonstrates complex contextual dependencies, (2) has greater semantic diversity, and (3) requires generalization to unseen domains due to its cross-domain nature and the unseen databases at test time. We experiment with two state-of-the-art text-to-SQL models adapted to the context-dependent, cross-domain setup. The best model obtains an exact match accuracy of 20.2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https://yale-lily.github.io/sparc.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Such domain shifts typically occur across related but different datasets (Torralba & Efros, 2011; Fang et al., 2013; Venkateswara et al., 2017; Yu et al., 2018b; 2019; Peng et al., 2019; Kamath et al., 2020), as well as from synthetic data to real data (including sim2real) (Wang et al., 2015; Ganin…”
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (2019, methodology): “We compare with the two baseline models released by Yu et al. (2019b).”
Task-Oriented Dialogue as Dataflow Synthesis (2020, background): “Here the prototypical problem is iterated question answering (Hemphill et al., 1990; Yu et al., 2019b), in which the user asks a sequence of questions corresponding to database queries, and results of query execution are presented as structured result sets.”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “…Collins, 2012; Oda et al., 2015; Wang et al., 2015; Yin et al., 2018), and with addition information such as database schemas (Zhong et al., 2017; Yu et al., 2018, 2019b,a) or input-output examples (Polosukhin and Skidanov, 2018; Zavershynskyi et al., 2018), synthesizing general-purpose code…”
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020, unknown): “The family of Spider datasets (Yu et al., 2018, 2019a,b) contain queries even more complex than in WTQ, including a higher percentage of nested queries and multiple table joins.”
Context Dependent Semantic Parsing: A Survey (2020, methodology): “Different from (Yu et al., 2019a), Vlachos and Clark (2014) integrated the dialogue acts into the MRs .”
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020, background): “…limited domains (Zelle and Mooney, 1996; Deborah A. Dahl and Shriber, 1994), or domain-specific languages with a limited set of operators (Berant et al., 2013; Quirk et al., 2015; Dong and Lapata, 2016; Liang et al., 2017; Krishnamurthy et al., 2017; Zhong et al., 2017; Yu et al., 2018, 2019b,a).”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “The majority of this work has focused on mapping a single query to the corresponding SQL with the exception of a few datasets, e.g., SParC (Yu et al., 2019b) and CoSQL (Yu et al., 2019a), that target inducing SQL parses for sequentially related questions.”
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases (2019, methodology): “To drive the progress of building a context-dependent NL query system, corpora such as ATIS (Hemphill et al., 1990; Dahl et al., 1994) and SParC (Yu et al., 2019)1 have been released.”
Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing (2021, methodology): “It employs a column attention mechanism to compute representations for the previous question and SQL. • CD-Seq2Seq (Yu et al. 2019b) is based on sequence-tosequence modelling extended with the turn-level history encoder proposed in (Suhr, Iyer, and Artzi 2018), which modifies the database schema…”
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2020, background): “Having a representation for such sub-trees can be useful when parsing the second question, in benchmarks such as SPARC (Yu et al., 2019).”
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, background): “They contrast the task with conversational semantic parsing (Suhr et al., 2018; Yu et al., 2019b,a; Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context.”
Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “In CoSQL (Yu et al., 2019a), users engage in a dialogue with a conversational database interface that is enacted by a SQL-competent wiz-ard.”
Semantic Evaluation for Text-to-SQL with Distilled Test Suite (2020, methodology): “We distill one shared test suite for the development set of S pider (Yu et al., 2018), C o SQL (Yu et al., 2019a) and SP ar C (Yu et al., 2019b), since they share the same 20 database schemata.”
Value-Agnostic Conversational Semantic Parsing (2021, background): “Recent work on context-dependent semantic parsing (e.g., Suhr et al., 2018; Yu et al., 2019) conditions on explicit representations of user utterances and programs with a neural encoder.”