Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
By Luyu Gao, Jamie Callan. 2021
395 citations recorded by the source corpus.
Recent research demonstrates the effectiveness of using fine-tuned language models (LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavily engineered fine-tuning pipelines to realize their full potential. In this paper, we identify and address two underlying problems of dense retrievers: i) fragility to training data noise and ii) requiring large batches to robustly learn the embedding space. We use the recently proposed Condenser pre-training architecture, which learns to condense information into the dense vector through LM pre-training. On top of it, we propose coCondenser, which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. Experiments on MS-MARCO, Natural Question, and Trivia QA datasets show that coCondenser removes the need for heavy data engineering such as augmentation, synthesis, or filtering, and the need for large batch training. It shows comparable performance to RocketQA, a state-of-the-art, heavily engineered system, using simple small batch fine-tuning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Sebastian Riedel
- Jianfeng Gao
- Graham Neubig
- Hannaneh Hajishirzi
- Jimmy J. Lin
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Mohit Bansal
In Information Systems
- Luyu Gao
- Wayne Xin Zhao
- Ji-rong Wen
- Jing Liu
- Ruiyang Ren
- Omar Khattab
- Jamie Callan
- Niklas Muennighoff
In Retrieval-Augmented Generation
- Sebastian Riedel
- Matei A. Zaharia
- Ruiyang Ren
- Jing Liu
- Luyu Gao
- Jamie Callan
- Edouard Grave
- Christopher Potts
In Large Language Models
- Jonathan Berant
- Jamie Callan
- Sebastian Riedel
- Luyu Gao
- Wayne Xin Zhao
- Ji-rong Wen
- Nils Reimers
- Niklas Muennighoff
Citations in the corpus
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Concurrent to this work, Gao & Callan (2021) have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure.”
- Learning to Retrieve Passages without Supervision (2021, methodology): “…on fine-tuning such retrievers on large datasets after pretraining (Lee et al., 2019; Chang et al., 2020; Guu et al., 2020; Sachan et al., 2021a; Gao and Callan, 2021a), we attempt to bridge the gap between unsupervised dense models and strong sparse (e.g. BM25; Robertson and Zaragoza 2009) or…”
- Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, neutral): “At the document level, text (chunk) encoders pre-trained with contrastive objectives learn to encode documentdocument similarity into inner products (Izacard et al., 2021; Gao and Callan, 2022).”
- Tool learning with large language models: a survey (2024, neutral): “This demonstrates accuracy and effectiveness of GPT-4 in tool selection.”
- Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “Most modern information retrieval systems are based on the index-then-retrieve pipeline where documents are first encoded and stored in an inverted index [10, 13, 15, 26, 34, 55, 58] or a nearest neighbor search index [7, 14, 20, 21, 25, 29, 49, 51] and search results are then constructed based on a”
- Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, background): “…tasks (Lee et al., 2019; Chang et al., 2020; Khattab & Zaharia, 2020; Izacard et al., 2022a; Oguz et al., 2022) or pre-training architectures (Gao & Callan, 2021; 2022), improving expressiveness using multi-vector representations (Luan et al., 2021), improving negative contrast (Qu et al.,…”
- Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “As dense unsupervised baselines, we consider the retriever from REALM (Guu et al., 2020), as well as RoBERTa large fine-tuned with SimCSE (Gao et al., 2021).”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “index 2021 Condenser [112] Condenser SR CE IB REP 2021 DRPQ [113] BERTbase MR NLL IB+DyHN Multiple pseudo query embeddings 2021 RANCE [114] RoBERTabase SR NLL DeHN 2021 DANCE [115] RoBERTabase SR IOC SHN Contrastive dual learning 2021 DPR-PAQ [116] RoBERTalarge SR NLL IB+SHN ALD GAP 2021 JPQ [117] B”
- Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “We totally used ∼ 3M pairs for fine-tuning, which is a combination of training data used by previous research (Gao et al., 2021; Gao and Callan, 2022; Asai et al., 2023; Su et al., 2023; Li et al., 2023).”
- Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “The ield has witnessed a drastic change in paradigm with the advent and wide availability of large-scale pretrained language models, such as BERT [46], GPT [21, 108], T5 [109] and BART [79], which are now pervasive in every high-performance system for Machine Translation [25], Question Answering [95”
- MTEB: Massive Text Embedding Benchmark (2022, methodology): “(a) Transformer encoder methods coCondenser (Gao and Callan, 2021), Contriever (Izacard et al.”
- Condenser: a Pre-training Architecture for Dense Retrieval (2021, background): “In a follow-up work, we show further improved performance adding contrastive learning to Condenser language model pretraining (Gao and Callan, 2021).”
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “, 2020a), and (3) improved pretraining (Gao and Callan, 2021).”
- A Systematic Multi-Domain Evaluation of Document Retrievers (2026, unknown): “Retrievers. docT5query [58] expands documents by generating multiple queries for each and retrieving with BM25. query2doc [80] instead expands the query, generating a pseudo-document and again retrieving with BM25, while HyDE [19] generates a hypothetical document and retrieves with the Contriever.”
- ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025, unknown): “We compare our ExpandR model with four representative retrieval models, including BM25 (Robertson et al., 2009), DPR (Karpukhin et al., 2020), CoCondenser (Gao and Callan, 2022), and ANCE (Xiong et al., 2021a).”