Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
By Yu Gu, Sue E. Kase, Michelle T. Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, Yu Su. 2020
329 citations recorded by the source corpus.
Existing studies on question answering on knowledge bases (KBQA) mainly operate with the standard i.i.d. assumption, i.e., training distribution over questions is the same as the test distribution. However, i.i.d. may be neither achievable nor desirable on large-scale KBs because 1) true user distribution is hard to capture and 2) randomly sampling training examples from the enormous space would be data-inefficient. Instead, we suggest that KBQA models should have three levels of built-in generalization: i.i.d., compositional, and zero-shot. To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64,331 questions, GrailQA, and provide evaluation settings for all three levels of generalization. In addition, we propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.1
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Caiming Xiong
- Percy Liang
- Graham Neubig
- Mohit Bansal
- Richard Socher
- Hannaneh Hajishirzi
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Hannaneh Hajishirzi
- Caiming Xiong
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Cordelia Schmid
In Question Answering & Reading Comprehension
- Huan Sun
- Xiang Deng
- Daniel Khashabi
- Michihiro Yasunaga
- Luke S. Zettlemoyer
- Jie Tang
- Caiming Xiong
- Hannaneh Hajishirzi
In Knowledge Graphs & Representation
- Harry Shum
- Noah A. Smith
- Minlie Huang
- Daniel Khashabi
- Michihiro Yasunaga
- Luke S. Zettlemoyer
- Jie Tang
- Hannaneh Hajishirzi
Citations in the corpus
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “, 2018) 512 32 1 GrailQA (Gu et al., 2021) 512 32 4 WebQSP (Yih et al.”
- UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, background): “The past few years has seen a lot of activity toward unified model designs (Friedman et al., 2021; Aghajanyan et al., 2021; Gupta et al., 2021; Tafjord and Clark, 2021; Lourie et al., 2021; Jiang et al., 2021; Aribandi et al., 2021; Zhong et al., 2021a; Chen et al., 2021; Zhong et al., 2022).”
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, methodology): “Note that, for two big datasets GrailQA and Simple Questions, we only randomly selected 1,000 samples each for testing in order to save computational cost.”
- RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2021, methodology): “Aside from the generalization challenge, G RAIL QA also presents additional dif-ficulty in terms of the large number of involved entities/relations, complex compositionality in the logical forms (up to 4 hops), and noisiness of the entities mentioned in questions (Gu et al., 2021).”
- A Survey of Large Language Models (2023, unknown): “0 [296], GrailQA [297], KQApro [298], CWQ [299], MKQA [300], ScienceQA [301]”
- Mind2Web: Towards a Generalist Agent for the Web (2023, background): “Many studies have centered around environments underpinned by a well-structured schema or ontology, including relational databases [32, 39] and knowledge bases [8, 38], which may not adequately reflect the more heterogeneous conditions in real-world situations.”
- AgentBench: Evaluating LLMs as Agents (2023, methodology): “In an effort to gauge the decision-making abilities of LLMs, specifically their proficiency in long-term planning, we have meticulously compiled a dataset sourced from preexisting knowledge base question answering (KBQA) datasets on FREEBASE, including GrailQA [28], ComplexWebQuestions [77], and Gra”
- ReTraCk: A Flexible and Efficient Framework for Knowledge Base Question Answering (2021, methodology): “Notably, both T RANSDUCTION and R ANKING models proposed by Gu et al. (2021) on GrailQA can be based on either GloVe (Pennington et al., 2014) or BERT (Devlin et al., 2019).”
- Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “Other methodologies involve initially training retrievers and generators separately, followed by joint training techniques to fine-tune both components.”