Harvesting Paragraph-level Question-Answer Pairs from Wikipedia

By Xinya Du, Claire Cardie. 2018

174 citations recorded by the source corpus.

We study the task of generating from Wikipedia articles question-answer pairs that cover content beyond a single sentence. We propose a neural network approach that incorporates coreference knowledge via a novel gating mechanism. As compared to models that only take into account sentence-level information (Heilman and Smith, 2010; Du et al., 2017; Zhou et al., 2017), we find that the linguistic knowledge introduced by the coreference representation aids question generation significantly, producing models that outperform the current state-of-the-art. We apply our system (composed of an answer span extraction system and the passage-level QG system) to the 10,000 top ranking Wikipedia articles and create a corpus of over one million question-answer pairs. We provide qualitative analysis for the this large-scale generated corpus from Wikipedia.

Subfields: Artificial Intelligence, Communication, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Yejin Choi
  5. Furu Wei
  6. Hannaneh Hajishirzi
  7. Percy Liang
  8. Graham Neubig

In Communication

  1. Hangbo Bao
  2. Mohit Bansal
  3. Nan Yang
  4. Patrick Lewis
  5. Li Dong
  6. Tristan Thrush
  7. Luis Enrico Lopez
  8. Yejin Choi

In Retrieval-Augmented Generation

  1. Luke S. Zettlemoyer
  2. Sebastian Riedel
  3. Douwe Kiela
  4. Jianfeng Gao
  5. Michael Lewis
  6. Hangbo Bao
  7. Mohit Bansal
  8. Nan Yang

In Natural Language Processing

  1. Mohit Bansal
  2. Yejin Choi
  3. Luke S. Zettlemoyer
  4. Jianfeng Gao
  5. Furu Wei
  6. Hannaneh Hajishirzi
  7. Percy Liang
  8. Douwe Kiela

Citations in the corpus