KILT: a Benchmark for Knowledge Intensive Language Tasks
By Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel and 1 more. 2020
820 citations recorded by the source corpus.
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Caiming Xiong
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Mohit Bansal
- Jianfeng Gao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Akari Asai
- Danqi Chen
- Jason Aaron Edward Weston
- Wen-tau Yih
- Matei A. Zaharia
- Christopher Potts
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Jianfeng Gao
- Sebastian Riedel
- Caiming Xiong
Citations in the corpus
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “As our search engine, we use a BM25 retriever (Robertson et al., 1995; Baeza-Yates et al., 1999) that indexes the Wikipedia dump from KILT (Petroni et al., 2021).”
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “Among those are KILT (Petroni et al., 2020) which focuses on knowledge-intensive tasks and retrieval-based models, Storium (Akoury et al., 2020) which focuses on open-ended story generation, and BIG bench3 which focuses on measuring few-shot and zero-shot capabilities of language models.”
- Improving Wikipedia verifiability with AI (2022, methodology): “We use Sphere (Piktus et al., 2021), a web-scale corpus and search infrastructure for web-scale data, as a source of candidate web pages.”
- Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, methodology): “…concern we need to raise is whether we are really improving the quality of the model, or simply moving the training set construction closer to the way the tests sets have been con- 2The leaderboard is available at: https://eval.ai/web/challenges/challenge-page/689 structed by Petroni et al. (2021).”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “Moreover, SEAL’s intersection formulation improves the state-of-the-art on passage-level retrieval by more than 10 points on the KILT benchmark (Petroni et al., 2021), contributing in establishing new state-of-the-art downstream re-sults on multiple datasets when paired with existing reader…”
- The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “We use KILT (Petroni et al., 2021), a standard KI-NLP benchmark, as well as a range of common sense tasks listed in Table 3 to evaluate our work.”
- Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, methodology): “We use the version of this dataset provided in the KILT benchmark (Petroni et al., 2021).”
- Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “We employ the standard KiLT Wikipedia dump (Petroni et al., 2020) as our knowledge source for retrieval for both datasets2.”
- Autoregressive Entity Retrieval (2020, methodology): “The idea of using a generative model for entity disambiguation was proposed in Petroni et al. (2020b) as they trained both BART and T5 in a seq2seq fashion on all KILT tasks (including ED).”
- Internet-Augmented Dialogue Generation (2021, methodology): “We use the pre-trained DPR model from the KILT Benchmark (Petroni et al., 2020).”
- Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, methodology): “We retain and supply all passage-relevant KILT metadata (such as IDs and provenances) with our collected datasets to facilitate future work.”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “KILT [49] is a benchmark that contains a suite of tasks/datasets for evaluating and analyzing knowledge-intensive NLP models.”
- Editing Factual Knowledge in Language Models (2021, methodology): “We evaluate on closed-book fact-checking (FC) fine-tune a BERT base model (Devlin et al., 2019) on the binary FEVER dataset (Thorne et al., 2018) from KILT (Petroni et al., 2021).”
- Modifying Memories in Transformer Models (2020, methodology): “Since the relations in different splits from KILT do not overlap, we construct the modification benchmark from only the training set of zsRE, and split the questions for each fact to obtain the training and test sets for mod-ification.”
- Towards Continual Knowledge Learning of Language Models (2021, methodology): “, 2019), which requires probing LMs for world knowledge in a zero-shot manner through slot-filling, and promising results utilizing the encoded world knowledge when finetuned on various Knowledge Intensive Language Tasks (KILT) (Petroni et al., 2021), e.”
- Hurdles to Progress in Long-form Question Answering (2021, neutral): “Baselines : We compare our model with the other entries on the ELI5 KILT leaderboard which are either generation-only, like T5-base (Raffel et al., 2020) and BART (Lewis et al., 2020b), or variants of BART using retrieval such as RAG (Lewis et al., 2020c) and BART + DPR (Petroni et al., 2020).”