KILT: a Benchmark for Knowledge Intensive Language Tasks

By Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel and 1 more. 2020

820 citations recorded by the source corpus.

Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Hannaneh Hajishirzi
  6. Yejin Choi
  7. Caiming Xiong
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Mohit Bansal
  5. Jianfeng Gao
  6. Xiaogang Wang
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Sebastian Riedel
  3. Akari Asai
  4. Danqi Chen
  5. Jason Aaron Edward Weston
  6. Wen-tau Yih
  7. Matei A. Zaharia
  8. Christopher Potts

In Natural Language Processing

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Hannaneh Hajishirzi
  5. Yejin Choi
  6. Jianfeng Gao
  7. Sebastian Riedel
  8. Caiming Xiong

Citations in the corpus