RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

By Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma. 2020

140 citations recorded by the source corpus.

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Replication Study of Dense Passage Retriever (2021, background): “…an important exemplar of text retrieval using learned dense representations, which has recently emerged as a promising alternative to “traditional” text retrieval using sparse bag-of-words representations (Zhan et al., 2020; Xiong et al., 2020; Hofstätter et al., 2020; Lin et al., 2020).”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “The method proposed in this paper, TILDE shares similarities with representation based methods [12, 16, 31], with two fundamental differences.”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “…directions: (1) distillation of more expressive architectures (Hofstätter et al., 2020; Lin et al., 2020) including explicit denoising (Qu et al., 2021; Ren et al., 2021b), (2) hard negative sampling (Xiong et al., 2020; Zhan et al., 2020a), and (3) improved pretraining (Gao and Callan, 2021).”
Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021, methodology): “BM25 [40] is a popular probabilistic BoW retrieval model that ranks documents based on the query terms appearing in each document.”
Shallow pooling for sparse labels (2021, background): “[43]) noticeably improves against runs with a similar MRR@10 on the original qrels.”
Optimizing Dense Retrieval Model Training with Hard Negatives (2021, methodology): “As we expect, random negative sampling methods, namely Rand Neg and In-Batch Neg, well minimize the total pairwise errors but cannot effectively minimize the top-K pairwise errors.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “index 2020 DPR [1] BERTbase SR NLL IB+SHN ALD 2020 ColBERT [95] BERTbase MR CE Multi-core pre-processing 2020 ME-BERT [96] BERTlarge MR CE IB+DyHN Sparse-dense hybrid 2020 RepBERT [97] BERTbase SR IOC IB 2020 ANCE [98] RoBERTabase SR NLL DyHN Async.”