Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval

By Luyu Gao, Jamie Callan. 2021

395 citations recorded by the source corpus.

Recent research demonstrates the effectiveness of using fine-tuned language models (LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavily engineered fine-tuning pipelines to realize their full potential. In this paper, we identify and address two underlying problems of dense retrievers: i) fragility to training data noise and ii) requiring large batches to robustly learn the embedding space. We use the recently proposed Condenser pre-training architecture, which learns to condense information into the dense vector through LM pre-training. On top of it, we propose coCondenser, which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. Experiments on MS-MARCO, Natural Question, and Trivia QA datasets show that coCondenser removes the need for heavy data engineering such as augmentation, synthesis, or filtering, and the need for large batch training. It shows comparable performance to RocketQA, a state-of-the-art, heavily engineered system, using simple small batch fine-tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Retrieval-Augmented Generation, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Mohit Bansal
  3. Percy Liang
  4. Sebastian Riedel
  5. Jianfeng Gao
  6. Graham Neubig
  7. Hannaneh Hajishirzi
  8. Jimmy J. Lin

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Andrew Zisserman
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Ming-Hsuan Yang
  8. Mohit Bansal

In Information Systems

  1. Luyu Gao
  2. Wayne Xin Zhao
  3. Ji-rong Wen
  4. Jing Liu
  5. Ruiyang Ren
  6. Omar Khattab
  7. Jamie Callan
  8. Niklas Muennighoff

In Retrieval-Augmented Generation

  1. Sebastian Riedel
  2. Matei A. Zaharia
  3. Ruiyang Ren
  4. Jing Liu
  5. Luyu Gao
  6. Jamie Callan
  7. Edouard Grave
  8. Christopher Potts

In Large Language Models

  1. Jonathan Berant
  2. Jamie Callan
  3. Sebastian Riedel
  4. Luyu Gao
  5. Wayne Xin Zhao
  6. Ji-rong Wen
  7. Nils Reimers
  8. Niklas Muennighoff

Citations in the corpus