Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback

By Carolin (Haas) Lawrence, Stefan Riezler. 2018

58 citations recorded by the source corpus.

Counterfactual learning from human bandit feedback describes a scenario where user feedback on the quality of outputs of a historic system is logged and used to improve a target system. We show how to apply this learning framework to neural semantic parsing. From a machine learning perspective, the key challenge lies in a proper reweighting of the estimator so as to avoid known degeneracies in counterfactual learning, while still being applicable to stochastic gradient optimization. To conduct experiments with human users, we devise an easy-to-use interface to collect human feedback on semantic parses. Our work is the first to show that semantic parsers can be improved significantly by counterfactual learning from logged human feedback data.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “…dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019).”
Learning to summarize from human feedback (2020, background): “Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68, 21], translation [32, 1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46].”
Recursively Summarizing Books with Human Feedback (2021, background): “…dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al.,…”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, methodology): “In the NLMAP(QT) setting, we split NLMapV2 into 4 tasks with queries in different types.”
Counterfactual Story Reasoning and Generation (2019, background): “5045 cognitive science (Byrne, 2002), as well as natural language processing (Hobbs, 2005; Lawrence and Riezler, 2018; Son et al., 2017).”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “Thus, RLHF has been positively applied to fine-tuning language models in most domains in natural language process (NLP) such as dialogue [88], [89], translation [90], [91], story generation [92], evidence extraction [93], [94], and semantic parsing [95].”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “, 2016), event generation (Zhou and Xu, 2020), semantic parsing (Lawrence and Riezler, 2019) and instruction following (Ouyang et al.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, methodology): “can be reweighted one-step-late (OSL) (Lawrence and Riezler, 2018) using θ′ from some previous iteration:”
TaPas: Weakly Supervised Table Parsing via Pre-training (2020, background): “, 2015), human in the loop (Iyer et al., 2017; Lawrence and Riezler, 2018) and training on examples from other domains (Herzig and Berant, 2017; Su and Yan, 2017).”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “Lawrence and Riezler (2018) show that a text-to-Overpass parser can be improved using historic logs of token-level binary feedback (collected using a graphical user interface that maps an Overpass query to predeﬁned blocks) on generated parses.”