Learning to Retrieve Passages without Supervision

By Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, Amir Globerson. 2021

68 citations recorded by the source corpus.

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our “recurring span retrieval” approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Izacard et al. (2022) explored diﬀerent contrastive learning methods to train retrievers, while Ram et al. (2022) used recurring spans within a document to create pseudo-positive query-document pairs.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Spider (Ram et al., 2021), a contemporary work, uses spans appearing multiple times in a document to create pseudo examples for contrastive learning in order to train unsupervised retrievers.”
In-Context Retrieval-Augmented Language Models (2023, methodology): “Adapting the use of a BM25 retriever (Robertson and Zaragoza, 2009) to the LM task (§5) yields significant gains, and choosing the grounding documents via our new class of Predictive Rerankers (§6) provides a further boost.”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, background): “A recent work in open domain question answering [Ram et al., 2021] used cross-passage recurring spans of text as the positives - one span act as the anchor (or ‘query’ in dense retrieval terms), while another acts as the positive.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “• Recurring Span Retrieval (RSR) [127]: RSR proposes to use recurring spans (i.”