Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

By Ahmed Elgohary, Saghar Hosseini, Ahmed Hassan Awadallah. 2020

81 citations recorded by the source corpus.

We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a corresponding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feedback to correct the system when it generates an inaccurate interpretation of an initial utterance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of utterances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the correction task and show that incorporating such a rich form of feedback can significantly improve the overall semantic parsing accuracy while retaining the flexibility of natural language interaction. While we estimated human correction accuracy is 81.5%, our best model achieves only 25.1%, which leaves a large gap for improvement in future research. SPLASH is publicly available at https://aka.ms/Splash_dataset.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Training Language Models with Language Feedback (2022, background): “Other work uses language feedback at test time to correct mistakes in a model’s behavior, for e.g. image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[56] Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah.”
Interscript: A dataset for interactive learning of scripts through error feedback (2021, background): “There are various modes of interaction (through labels (Raghavan 2006; Fails and Olsen 2003), utterance (Radlinski et al. 2019), imitation (Brantley, Sharaf, and Daum’e 2020), and language (Elgohary, Awadallah et al. 2020)).”
Training Language Models with Natural Language Feedback (2022, background): “Other work uses language feed-back at test time to correct mistakes in a model’s behavior, for e.g. image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
Teaching Large Language Models to Self-Debug (2023, methodology): “Feedback: With the above function, square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100].”
Improving Code Generation by Training with Natural Language Feedback (2023, background): “Elgohary et al. (2020); Austin et al. (2021); Nijkamp et al. (2022) all demonstrate that code LLM performance generally improves when prompted with natural language feed-back, though Nijkamp et al. (2022) observes that the feed-back is more effective when it is given one step at a time.”
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “(Nijkamp et al., 2023; Elgohary et al., 2020), iterative output refinement (Saunders et al., 2022), and creative writing (Lee et al., 2022a; Shu et al., 2023; Wang et al., 2023b), generative information-seeking (Kamalloo et al., 2023), and even theorem proving (Yang et al., 2023b).”
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, methodology): “For example, the SQL explanations of (Elgohary et al., 2020) express “group by” in simple language “for each vote_id, find .”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “By using a self-debugging prompting approach [44], which contains simple feedback, unit-test, and code explanation prompts module, the text-to-SQL [122] model is able to provide a solution it states is correct or a maximum number of attempts has been made.”
Training Language Models with Language Feedback at Scale (2023, background): “Other work uses language feedback at test time to correct mistakes in a model’s behavior, e.g., image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
Context Dependent Semantic Parsing: A Survey (2020, methodology): “Most Feedback Semantic Parsing systems (Iyer et al., 2017; Yao et al., 2019b; Yao et al., 2019a; Elgohary et al., 2020) start with using an C I SP parser to parse a given utterance into an initial MR .”