A Replication Study of Dense Passage Retriever

By Xueguang Ma, Kai Sun, Ronak Pradeep, Jimmy J. Lin. 2021

74 citations recorded by the source corpus.

Text retrieval using learned dense representations has recently emerged as a promising alternative to"traditional"text retrieval using sparse bag-of-words representations. One recent work that has garnered much attention is the dense passage retriever (DPR) technique proposed by Karpukhin et al. (2020) for end-to-end open-domain question answering. We present a replication study of this work, starting with model checkpoints provided by the authors, but otherwise from an independent implementation in our group's Pyserini IR toolkit and PyGaggle neural text ranking library. Although our experimental results largely verify the claims of the original paper, we arrived at two important additional findings that contribute to a better understanding of DPR: First, it appears that the original authors under-report the effectiveness of the BM25 baseline and hence also dense--sparse hybrid retrieval results. Second, by incorporating evidence from the retriever and an improved answer span scoring technique, we are able to improve end-to-end question answering effectiveness using exactly the same models as in the original work.

Subfields: Artificial Intelligence, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Here, our model is competitive with a strong BM25 baseline (Ma et al., 2021), for example leading to 3 points improvement for the recall@100 on NaturalQuestions.”
Learning to Retrieve Passages without Supervision (2021, methodology): “Thus, we define: shybrid(q, p) = s(q, p) + BM25(q, p) We adopt the normalization technique from Ma et al. (2021).”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “This is a standard approach and Pyserini adopts the specific implementation in TCT-ColBERT [17], but similar techniques are used elsewhere as well [13, 18].”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Here, our model is also competitive with a strong BM25 baseline (Ma et al., 2021), for example leading to 3 points improvement for the recall@100 on NaturalQuestions.”
Improving Passage Retrieval with Zero-Shot Question Generation (2022, background): “Prior work (Ma et al., 2021) has shown that BM25 is a strong baseline for the datasets we consider.”
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2021, methodology): “For instance, new 066 state-of-the-art performance on ODQA can be ob- 067 tained using a hybrid system of DPR and a simple 068 well-tuned BM25 (Ma et al., 2021).”
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021, methodology): “This strategy is similar to the one described by Ma et al. (2021).”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, background): “Many papers (Luan et al., 2021; Gao et al., 2020b; Ma et al., 2021; Lin et al., 2021) have demonstrated that sparse retrieval can complement dense retrieval via a simple linear combination of their scores.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “(Gao et al., 2021b,a; Ma et al., 2021; Lin and Ma, 2021; Cormack et al., 2009).”