Human-in-the-Loop Parsing

By Luheng He, Julian Michael, Michael Lewis, Luke S. Zettlemoyer. 2016

32 citations recorded by the source corpus.

This paper demonstrates that it is possible for a parser to improve its performance with a human in the loop, by posing simple questions to non-experts. For example, given the first sentence of this abstract, if the parser is uncertain about the subject of the verb “pose,” it could generate the question What would pose something? with candidate answers this paper and a parser. Any fluent speaker can answer this question, and the correct answer resolves the original uncertainty. We apply the approach to a CCG parser, converting uncertain attachment decisions into natural language questions about the arguments of verbs. Experiments show that crowd workers can answer these questions quickly, accurately and cheaply. Our human-in-the-loop parser improves on the state of the art with less than 2 questions per sentence on average, with a gain of 1.7 F1 on the 10% of sentences whose parses are changed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, background): “Recent work has focused on cheaply eliciting quality annotations from nonexperts through crowdsourcing (He et al., 2016; Iyer et al., 2017; Michael et al., 2018).”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, unknown): “He et al. (2016) ask simpliﬁed questions about uncertain dependencies in CCG parses and use the answers as soft constraints to regenerate the parse.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, methodology): “…of using the grammar as a constraint , with an eye towards enabling bootstrapping through human-in-the-loop semantic parsing, where humans quickly annotate data by manually correcting parses from an initial prototype (Duan et al., 2016; He et al., 2016; Yao et al., 2019; Elgohary et al., 2021).”
A Challenge Set and Methods for Noun-Verb Ambiguity (2018, background): “He et al. (2016) then showed that incorporating crowd-sourced annotations improves parsing by a noticeable margin on the subset of sentences in”
Large-Scale QA-SRL Parsing (2018, methodology): “Crowdsourcing has also been used for indirectly annotating syntax (He et al., 2016; Duan et al., 2016), and to complement expert annotation of SRL (Wang et al., 2018).”
Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study (2019, background): “We extend the NLG lexicon and grammar (Section 4.3) to accommodate this complexity, with details shown in Appendix A.”
Crowdsourcing Question-Answer Meaning Representations (2017, methodology): “…applied to gather annotations of structure in the setup of multiple choice questions, for example, to recover Dowty’s semantic proto-roles (Reisinger et al., 2015; White et al., 2016) and for human-in-the-loop parsing and classiﬁcation (He et al., 2016; Duan et al., 2016; Werling et al., 2015).”