The Value of Semantic Parse Labeling for Knowledge Base Question Answering
By Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, Jina Suh. 2016
725 citations recorded by the source corpus.
We demonstrate the value of collecting semantic parse labels for knowledge base question answering. In particular, (1) unlike previous studies on small-scale datasets, we show that learning from labeled semantic parses significantly improves overall performance, resulting in absolute 5 point gain compared to learning from answers, (2) we show that with an appropriate user interface, one can obtain semantic parses with high accuracy and at a cost comparable or lower than obtaining just answers, and (3) we have created and shared the largest semantic-parse labeled dataset to date in order to advance research in question answering.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Caiming Xiong
- Graham Neubig
- Hannaneh Hajishirzi
- Yejin Choi
In Computer Vision and Pattern Recognition
- Mohit Bansal
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Cordelia Schmid
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jonathan Berant
- Caiming Xiong
- Hannaneh Hajishirzi
- Richard Socher
- Yejin Choi
In Question Answering & Reading Comprehension
- William W. Cohen
- Haitian Sun
- Jonathan Berant
- Ashish Sabharwal
- Peter Clark
- Percy Liang
- Wen-tau Yih
- Yoav Goldberg
Citations in the corpus
- Break It Down: A Question Understanding Benchmark (2020, methodology): “, 2018), SPARQL for graph KBs (Yih et al., 2016), while other ad-hoc languages are used based on the task at hand.”
- Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “For example, (Yih et al., 2016) designed a user interface to help Freebase experts to efficiently create queries.”
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “For most semantic parsing tasks, we report the exact match accuracy of logical forms, and for task has test suite (Zhong et al., 2020), we add test suite metric to represent model’s performance; an exception is WebQSP, for which we follow previous work to execute the parses and report the F1 score.”
- The Web as a Knowledge-Base for Answering Complex Questions (2018, methodology): “To generate complex questions we use the dataset WEBQUESTIONSSP (Yih et al., 2016), which contains 4,737 questions paired with SPARQL queries for Freebase (Bollacker et al.”
- Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (2016, methodology): “l forms), or have employed synthetic datasets. We apply NSM to learn a semantic parser with weak supervision and no manual engineering. we used the challenging semantic parsing dataset WEBQUESTIONSSP [18], which consists of 3,098 question-answer pairs for training and 1,639 for testing. These questi”
- Search-based Neural Structured Learning for Sequential Question Answering (2017, background): “However, while having full supervision leads to a better semantic parser, collecting the correct parses requires a much more sophisticated UI design (Yih et al., 2016).”
- Learning a Neural Semantic Parser from User Feedback (2017, background): “Each of these schemes presents a particular trade-off between annotation effort and parser accuracy; however, recent work has suggested that labeled queries are the most effective (Yih et al., 2016).”
- Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, methodology): “…2(a) summarizes the overall statistics of CFQ. Table 2(b) uses numbers from (Finegan-Dollak et al., 2018) and from an analysis of WebQuestionsSP (Yih et al., 2016) and ComplexWebQuestions (Talmor & Berant, 2018) to compare three key statistics of CFQ to other semantic parsing datasets (none of…”
- Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems (2016, background): “Recently, Yih et al. (2016) showed that labeled semantic parses for the knowledge based question answering task can be obtained at a cost comparable to obtaining answers.”
- Improved Neural Relation Detection for Knowledge Base Question Answering (2017, methodology): “WebQSP (WQ): A multi-relation KBQA task.”
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020, background): “…of the XXXI Olympiad (Portuguese : Jogos da XXXI Olimpíada) and commonly known as Rio 2016 , was an international multi-sport event …… Be-rant et al., 2013), WebQSP (Yih et al., 2016), Web-Complex (Talmor and Berant, 2018), MetaQA (Zhang et al., 2018), WikiTableQuestion (Pasupat and Liang, 2015).”
- QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships (2018, background): “(A) the one to the west (B) the one to the north LF: qrel(loudness, higher, world1) → qrel(distance, lower, world2) ; qrel(distance, lower, world1) Yih et al. 2016).”
- Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, neutral): “Last, the semantics are more complex comparing to previous datasets like W EB Q UESTIONS SP [62].”
- Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation (2019, background): “The insight behind performing schema linking is partly inspired by the success of incorporating entity linking in knowledge base question answering and semantic parsing (Yih et al., 2016; Krishnamurthy et al., 2017; Yu et al., 2018a; Herzig and Berant, 2018; Kolitsas et al., 2018).”
- StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023, methodology): “, WebQuestionsSP (WebQSP) (Yih et al., 2016) and MetaQA (Zhang et al.”
- RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2021, background): “Past work has collected a series of datasets (Yih et al., 2016; Bor-des et al., 2015; Zhang et al., 2018; Su et al., 2016; Gu et al., 2021) as well as proposed a diversity of approaches for this task.”
- Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing (2018, methodology): “Furthermore, we want to collect labeled data by designing an interactive UI for annotation assist like (Yih et al., 2016), which uses semantic graphs to annotate the meaning of sentences, since semantic graph is more natural and can be easily annotated without the need of expert knowledge.”
- Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (2018, methodology): “Using WikiMovies (Miller et al., 2016) and WebQuestionsSP (Yih et al., 2016), we construct datasets with a varying amount of training supervision and KB completeness, and with a varying degree of question complexity.”