Domain-matched Pre-training Tasks for Dense Retrieval

By Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta and 1 more. 2021

72 citations recorded by the source corpus.

Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks. A notable exception is information retrieval, where additional pre-training has so far failed to produce convincing results. We show that, with the right pre-training setup, this barrier can be overcome. We demonstrate this by pre-training large bi-encoder models on 1) a recently released set of 65 million synthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Improving Wikipedia verifiability with AI (2022, methodology): “The dense retrieval sub-system is a neural network which learns from Wikipedia data to encode the citation context into a dense query vector (Wu et al., 2019; Karpukhin et al., 2020; Maillard et al., 2021; Oğuz et al., 2021; Luan et al., 2021).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “We train DPR WEB by ﬁnetuning a PAQ-based (Lewis et al., 2021) bi-encoder check-point (O˘guz et al., 2021) for 40 epochs on 16 GPUs.”
You Only Need One Model for Open-domain Question Answering (2021, background): “…two new * Work done while at Samsung Research lines of approach; 1) referring to internal knowledge parameterized in the model (Brown et al., 2020; Petroni et al., 2019; Roberts et al., 2020), and 2) referring to external knowledge retrieved by matching query and knowledge in dense representation…”
Learning to Retrieve Passages without Supervision (2021, methodology): “A concurrent work by Oğuz et al. (2021) presented DPR-PAQ, which shows strong results on NQ after pretraining.”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, unknown): “, 2020a, 2021b), and (3) improved pretraining (Gao and Callan, 2021; Oğuz et al., 2021).”
Boosted Dense Retriever (2021, background): “Recently, it has become common practice to run a number of rounds of dense retrieval training to bootstrap hard negatives (Xiong et al., 2020; Qu et al., 2021; O ˘ guz et al., 2021; Sachan et al., 2021, inter alia.).”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, unknown): “Recent research has primarily focused on the following aspects: developing better pre-training tasks (Lee et al., 2019; Chang et al., 2020; Izacard et al., 2022a; Gao & Callan, 2021; Oguz et al., 2022), improving contrastive negatives (Qu et al.”
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training (2021, background): “(2020), as well as the recently enhanced version (Oğuz et al., 2022).”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, methodology): “A concurrent work DPR-PAQ (O˘guz et al., 2021) revisits pre-training and prosposes domain matched pre-training, where they use a 65-million-size synthetic QA pair dataset generated with pre-trained Natural Question and Trivia QA pipelines to pre-train dense retrievers.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “Since Reddit data is no longer free available, we use two pre-processed version by sentence-transformers 4 and Oguz et al. (2022) for pair mining.”
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, background): “Dense Retrieval Models Dense retrieval models can be categorized into two groups, passage-level retrievers (Karpukhin et al., 2020; Oguz et al., 2021; Xiong et al., 2021; Gao and Callan, 2022) and token/phrase-level retrievers (Khattab and Za-haria, 2020; Khattab et al., 2020; Gao et al., 2021a;…”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “index 2021 Condenser [112] Condenser SR CE IB REP 2021 DRPQ [113] BERTbase MR NLL IB+DyHN Multiple pseudo query embeddings 2021 RANCE [114] RoBERTabase SR NLL DeHN 2021 DANCE [115] RoBERTabase SR IOC SHN Contrastive dual learning 2021 DPR-PAQ [116] RoBERTalarge SR NLL IB+SHN ALD GAP 2021 JPQ [117] B”