Learning a Neural Semantic Parser from User Feedback

By Srini Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke S. Zettlemoyer. 2017

377 citations recorded by the source corpus.

We present an approach to rapidly and easily build natural language interfaces to databases for new domains, whose performance improves over time based on user feedback, and requires minimal intervention. To achieve this, we adapt neural sequence models to map utterances directly to SQL with its full expressivity, bypassing any intermediate meaning representations. These models are immediately deployed online to solicit feedback from real users to flag incorrect queries. Finally, the popularity of SQL facilitates gathering annotations for incorrect predictions using the crowd, which is directly used to improve our models. This complete feedback loop, without intermediate representations or database specific engineering, opens up new ways of building high quality semantic parsers. Experiments suggest that this approach can be deployed quickly for any new target domain, as we show by learning a semantic parser for an online academic database from scratch.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, background): “Recent work has focused on cheaply eliciting quality annotations from nonexperts through crowdsourcing (He et al., 2016; Iyer et al., 2017; Michael et al., 2018).”
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017, neutral): “Iyer et al. (2017)”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, background): “Data Splits For the ﬁnal training dataset, we also select and include 752 queries and 1659 questions that follow our annotation protocol from six existing datasets: Restaurants, GeoQuery, Scholar, Academic, IMDB, and Yelp.”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, background): “Feedback semantic parsing interactively collects data from user feedbacks as continuous data streams but does not address the problem of catastrophic forgetting or improve forward transfer (Iyer et al., 2017; Yao et al., 2019; Labutov et al., 2018).”
Good-Enough Compositional Data Augmentation (2019, neutral): “Negative results We conclude with a corresponding set of experiments on the SCHOLAR text-toSQL dataset of Iyer et al. (2017), which is similar”
NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System (2018, background): “Others (Kwiatkowski et al., 2013; Long et al., 2016; Guu et al., 2017; Iyer et al., 2017; Zhong et al., 2017) evaluate the generated code through execution.”
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation (2019, methodology): “Existing tasks for mapping NL to source code primarily use a single NL utterance (Zettlemoyer and Collins, 2005; Iyer et al., 2017) to generate database queries (semantic parsing), single line python code (Yin et al.”
Coarse-to-Fine Decoding for Neural Semantic Parsing (2018, methodology): “, 2017), sharing parameters for multiple languages or meaning representations (Susanto and Lu, 2017; Herzig and Berant, 2017), and utilizing user feedback signals (Iyer et al., 2017).”
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2019, background): “As described by Yu et al., Spider contains 8,659 examples (questions and SQL queries, with the accompanying schemas), including 1,659 examples lifted from the Restaurants (Popescu et al., 2003; Tang and Mooney, 2000), GeoQuery (Zelle and Mooney, 1996), Scholar (Iyer et al., 2017), Academic (Li and J”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, methodology): “29 Iyer et al. (2018) 7.”
Natural Language to Structured Query Generation via Meta-Learning (2018, background): “Semantic Parsing Mapping natural language to logic forms has been actively studied in natural language processing research (Zettlemoyer and Collins, 2005; Giordani and Moschitti, 2010; Artzi and Zettlemoyer, 2011; Berant et al., 2013; Vlachos and Clark, 2014; Yih et al., 2014, 2015; Wang et al., 201”
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, background): “At the same time, related benchmarks are becoming increasingly complex, from the single-domain benchmarks such as ATIS (Iyer et al. 2017) and GeoQuery (Zelle and Mooney 1996) to the cross-domain benchmarks such as WikiSQL (Zhong, Xiong, and Socher 2017) and Spider (Yu et al. 2018c).”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, methodology): “…we build the PAGSAS benchmark, which consists of 5 sub-tasks and 124K examples from Advising (Finegan-Dollak et al., 2018), GeoQuery (Zelle and Mooney, 1996a), Scholar (Iyer et al., 2017), ATIS (Price, 1990; Dahl et al., 1994) and Spider (Yu et al., 2018) (Examples are shown in Table 1).”
Robust Text-to-SQL Generation with Execution-Guided Decoding (2018, methodology): “Iyer et al. (2017) apply a relatively standard sequenceto-sequence model to text-to-SQL tasks.”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, methodology): “Numerous prior works (Berant & Liang, 2014; Wang et al., 2015a; Jia & Liang, 2016; Iyer et al., 2017; Andreas, 2020) have demonstrated the benefits of augmenting data using context-free grammar.”
Learning from On-Line User Feedback in Neural Question Answering on the Web (2019, background): “Applications in these disciplines include user feedback that is leveraged in order to improve conversational agents [36], assess the relevance of answers in search queries for knowledge graphs [39], or identify incorrect queries to databases [18].”
Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases (2019, background): “Despite recent efforts to synthesize NLQ-SQL pairs [18], [38] or derive them from user descriptions of SQL queries [9], obtaining realistic labeled data remains an open research challenge.”
TaPas: Weakly Supervised Table Parsing via Pre-training (2020, background): “, 2015), human in the loop (Iyer et al., 2017; Lawrence and Riezler, 2018) and training on examples from other domains (Herzig and Berant, 2017; Su and Yan, 2017).”
SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning (2017, background): “Recent work considers mitigating this issue by incorporating users’ guidance (Li & Jagadish, 2014; Iyer et al., 2017).”