Semantic Parsing on Freebase from Question-Answer Pairs

By Jonathan Berant, Andrew K. Chou, Roy Frostig, Percy Liang. 2013

2,231 citations recorded by the source corpus.

In this paper, we train a semantic parser that scales up to Freebase. Instead of relying on annotated logical forms, which is especially expensive to obtain at large scale, we learn from question-answer pairs. The main challenge in this setting is narrowing down the huge number of possible logical predicates for a given question. We tackle this problem in two ways: First, we build a coarse mapping from phrases to predicates using a knowledge base and a large text corpus. Second, we use a bridging operation to generate additional predicates based on neighboring predicates. On the dataset of Cai and Yates (2013), despite not having annotated logical forms, our system outperforms their state-of-the-art parser. Additionally, we collected a more realistic and challenging dataset of question-answer pairs and improves over a natural baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple and Effective Multi-Paragraph Reading Comprehension (2017, background): “Knowledge bases can be used, such as in (Berant et al., 2013), although the resulting systems are limited by the quality of the knowledge base.”
Language Models are Few-Shot Learners (2020, neutral): “rk suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being i”
PaLM: Scaling Language Modeling with Pathways (2022, background): “Next we study the variance of few-shot learning performance on the StoryCloze, Natural Questions, and Web Questions benchmarks presented in Section 6.1.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Our RAG models achieve state-of-the-art results on open Natural Questions [25], WebQuestions [3] and CuratedTrec [2] and strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [20].”
Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “WebQuestions (WQ) (Berant et al., 2013) consists of questions selected using Google Suggest API, where the answers are entities in Freebase.”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “WebQuestions The WebQuestions dataset (Berant et al., 2013) was collected from the Google Suggest API, using one seed question and expanding the set to related questions.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “Question Answering We look at Web Questions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “…(Kwiatkowski et al., 2019), a dataset of questions from web queries, each accompanied by a Wikipedia article containing the answer; We-bQuestions (Berant et al., 2013), comprising questions from web queries matched to corresponding entries in FreeBase (Bollacker et al., 2008); and TriviaQA…”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, methodology): “To train our model, we adopt the weakly supervised training method widely used in the semantic parsing literature (Berant et al., 2013a).”
CoQA: A Conversational Question Answering Challenge (2018, background): “Another common knowledge source is machine-friendly databases, which organize world facts in the form of a table or a graph (Berant et al., 2013; Pasupat and Liang, 2015; Bordes et al., 2015; Saha et al., 2018; Talmor and Berant, 2018).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Open-Domain Question Answering: TriviaQA (Joshi et al., 2017), Natural Questions (NQS) (Kwiatkowski et al., 2019), Web Questions (WebQS) (Berant et al., 2013) Cloze and Completion Tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al.,…”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “This is commonly used in structured generation problems such as semantic parsing, program synthesis and molecule generation (Liang et al., 2013; Berant et al., 2013; Guu et al., 2017; Min et al., 2019; Zhong et al., 2020; Yang et al., 2020).”
SPoC: Search-based Pseudocode to Code (2019, background): “One of its traditional tasks is to parse a given question (usually a single sentence) into an executable database query [42, 43, 20, 4, 44, 39].”
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems (2017, background): “som, 2013), and inherits ideas from the extensive literature on semantic parsing (Jones et al., 2012; Berant et al., 2013; Andreas et al., 2013; Quirk et al., 2015; Liang et al., 2016; Neelakantan et al., 2016) and program generation (Reed and de Freitas, 2016; Graves et al.”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “Setup TriviaQA and WebQuestions were introduced as reading comprehension and semantic parsing tasks, respectively (Joshi et al., 2017; Berant et al., 2013).”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, methodology): “On datasets where question writers do not know the answer— Natural Questions (Kwiatkowski et al., 2019), WebQuestions (Berant et al., 2013), and Curat-edTrec (Baudis and Sediv´y, 2015)—we show that learned retrieval is crucial, providing improvements of 6 to 19 points in exact match over BM25.”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, background): “The task is to produce the answer to a question by exploiting resources such as documents (Voorhees, 1999), webpages (Kwok et al., 2001) or structured knowledge bases (Berant et al., 2013; Bordes et al., 2015; Yu et al., 2017).”