Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases

By Yu Gu, Sue E. Kase, Michelle T. Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, Yu Su. 2020

329 citations recorded by the source corpus.

Existing studies on question answering on knowledge bases (KBQA) mainly operate with the standard i.i.d. assumption, i.e., training distribution over questions is the same as the test distribution. However, i.i.d. may be neither achievable nor desirable on large-scale KBs because 1) true user distribution is hard to capture and 2) randomly sampling training examples from the enormous space would be data-inefficient. Instead, we suggest that KBQA models should have three levels of built-in generalization: i.i.d., compositional, and zero-shot. To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64,331 questions, GrailQA, and provide evaluation settings for all three levels of generalization. In addition, we propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Caiming Xiong
  3. Percy Liang
  4. Graham Neubig
  5. Mohit Bansal
  6. Richard Socher
  7. Hannaneh Hajishirzi
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Hannaneh Hajishirzi
  3. Caiming Xiong
  4. Dacheng Tao
  5. Trevor Darrell
  6. Ming-Hsuan Yang
  7. Xiaogang Wang
  8. Cordelia Schmid

In Question Answering & Reading Comprehension

  1. Huan Sun
  2. Xiang Deng
  3. Daniel Khashabi
  4. Michihiro Yasunaga
  5. Luke S. Zettlemoyer
  6. Jie Tang
  7. Caiming Xiong
  8. Hannaneh Hajishirzi

In Knowledge Graphs & Representation

  1. Harry Shum
  2. Noah A. Smith
  3. Minlie Huang
  4. Daniel Khashabi
  5. Michihiro Yasunaga
  6. Luke S. Zettlemoyer
  7. Jie Tang
  8. Hannaneh Hajishirzi

Citations in the corpus