Multi-Task Retrieval for Knowledge-Intensive Tasks

By Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oğuz, Veselin Stoyanov, Gargi Ghosh. 2021

69 citations recorded by the source corpus.

Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be _universal_ and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Representation & Self-Supervised Learning

Citations in the corpus

Improving Wikipedia verifiability with AI (2022, methodology): “The dense retrieval sub-system is a neural network which learns from Wikipedia data to encode the citation context into a dense query vector (Wu et al., 2019; Karpukhin et al., 2020; Maillard et al., 2021; Oğuz et al., 2021; Luan et al., 2021).”
Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, background): “In their initial baseline setup Petroni et al. (2021) already studied the impact of multi-task retrieval training; Maillard et al. (2021) continued to study various configurations for KILT multi-task single-model retrieval.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “Moreover, SEAL’s intersection formulation improves the state-of-the-art on passage-level retrieval by more than 10 points on the KILT benchmark (Petroni et al., 2021), contributing in establishing new state-of-the-art downstream re-sults on multiple datasets when paired with existing reader…”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “In line with previous research (Maillard et al., 2021; O˘guz et al., 2021) we also note that zero shot transfer of DPR models doesn’t yield good results, leaving the challenge of building universal, neural web retrievers open.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “Few-shot and zero-shot retrieval will also be of increasing importance, and there are already works investigating this direction (Maillard et al., 2021; Thakur et al., 2021).”
Robust Retrieval Augmented Generation for Zero-shot Slot Filling (2021, methodology): “, 2020b) used DPR pre-trained on Natural Questions, and Multi-DPR (Maillard et al., 2021) trained on all KILT tasks jointly.”
Task-aware Retrieval with Instructions (2022, methodology): “Instructions provide extra flexibility and enable zero-shot transfer via natural language instructions, unlike training with fixed task tags (Maillard et al., 2021).”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “On the FEVER hidden test set, 8 our model yields 88.5% down-stream accuracy and ranks second among all submissions, outperforming all of prior published work (Maillard et al., 2021; Petroni et al., 2021; Lewis et al., 2020b).”
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2020, background): “This setting was previously investigated in several works (Karpukhin et al., 2020; Maillard et al., 2021; Qi et al., 2021), but not in the multi-source context.”
LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “Furthermore, we also highlight open challenges and future directions, shedding light on key limitations and potential research opportunities in LLM-based embedding techniques.”