Condenser: a Pre-training Architecture for Dense Retrieval
By Luyu Gao, Jamie Callan. 2021
308 citations recorded by the source corpus.
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs’ internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Furu Wei
- Mohit Bansal
- Sebastian Riedel
- Wen-tau Yih
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
In Computer Vision and Pattern Recognition
- Furu Wei
- Sebastian Riedel
- Wen-tau Yih
- Luc Van Gool
- Luke S. Zettlemoyer
- Dacheng Tao
- Jonathan Berant
- Jianfeng Gao
In Retrieval-Augmented Generation
- Sebastian Riedel
- Luyu Gao
- Wen-tau Yih
- Jamie Callan
- Fabio Petroni
- Jonathan Berant
- Ruiyang Ren
- Jing Liu
In Information Retrieval & Search
Citations in the corpus
- Learning to Retrieve Passages without Supervision (2021, background): “While most works focus on fine-tuning such retrievers on large datasets after pretraining (Lee et al., 2019; Chang et al., 2020; Guu et al., 2020; Sachan et al., 2021; Gao and Callan, 2021a), we attempt to bridge the gap between unsuper-”
- BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, methodology): “Considering the discreteness of tokens, we further introduce a patch aggregation strategy which explicitly encourages the [CLS] token to associate all patches (Gao & Callan, 2021).”
- C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “Particularly, it needs embedding-oriented pre-training to prepare the text encoder [16], contrastive learning with sophisticated negative sampling to improve the embedding’s discriminability [43], and instruction-based fine-tuning [7, 50] to integrate different representation capabilities of text…”
- Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “Later works studied the second stage pre-training of language models specifically for retrieval (Izacard et al., 2021; Gao and Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and Shao, 2022) as well as model scaling (Ni et al.”
- Boosted Dense Retriever (2021, background): “, and/or proposing pre-training objectives (O˘guz et al., 2021; Guu et al., 2020; Chang et al., 2020; Sachan et al., 2021; Gao and Callan, 2021).”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “ReACT [172] uses Chain-of-Thought (CoT) [173] to break queries down for recursive retrieval and provide richer information.”
- Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, methodology): “As a workaround, we employ GradCache (Luyu Gao and Callan, 2021) as well as mixed precision training (Micikevicius et al., 2018).”
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, methodology): “In this paper, we adopt a special pre-training architecture, Condenser (Gao and Callan, 2021).”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Furthermore, co-Condenser [78] extends the Condenser architecture by incorporating a query-agnostic contrastive loss based on the retrieval corpus, which pulls close the text segments from the same document while pushing away other unrelated segments.”
- Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, background): “…retrieval, significant research is dedicated to enhancing the text representation capabilities of pre-trained language models through the design of auxiliary pre-training tasks (Gao and Callan, 2021; Xiao et al., 2022; Gao and Callan, 2022; Wang et al., 2022a; Long et al., 2022b; Li et al., 2023).”
- C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “Particularly, it needs embedding-oriented pre-training to prepare the text encoder (Gao and Callan, 2021), contrastive learning with sophisticated negative sampling to improve the embedding’s discriminability (Qu et al., 2020), and instruction-based fine-tuning (Su et al., 2022; Asai et al., 2022)…”
- Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, background): “…tasks (Lee et al., 2019; Chang et al., 2020; Khattab & Zaharia, 2020; Izacard et al., 2022a; Oguz et al., 2022) or pre-training architectures (Gao & Callan, 2021; 2022), improving expressiveness using multi-vector representations (Luan et al., 2021), improving negative contrast (Qu et al.,…”
- Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “In contrast to sparse lexical methods, dense retrieval [61, 304, 314] uses continuous vector embeddings to represent queries and documents.”
- DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Works such as Condenser (Gao and Callan, 2021), RetroMAE (Xiao et al., 2022), and SimLM (Wang et al., 2023) have enhanced the dense representation of BERT via customized architectures during language modeling.”
- ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025, unknown): “Given a query q and a document collection D = { d 1 , ..., d k } , dense retrieval models (Karpukhin et al., 2020; Xiong et al., 2021a; Gao and Callan, 2021) first encode the query q and the i -th document d i into embeddings ⃗q and ⃗d i us-ing PLMs, such as BERT (Devlin et al., 2019): Then the…”
- LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “A straightforward approach to improve domain-specific performance is to do continual pre-training over the corpus in the target domain [40, 51].”