Optimizing Dense Retrieval Model Training with Hard Negatives

By Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, M. Zhang, Shaoping Ma. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021

352 citations recorded by the source corpus.

Ranking has always been one of the top concerns in information retrieval researches. For decades, the lexical matching signal has dominated the ad-hoc retrieval process, but solely using this signal in retrieval may cause the vocabulary mismatch problem. In recent years, with the development of representation learning techniques, many researchers turn to Dense Retrieval (DR) models for better ranking performance. Although several existing DR models have already obtained promising results, their performance improvement heavily relies on the sampling of training examples. Many effective sampling strategies are not efficient enough for practical usage, and for most of them, there still lacks theoretical analysis in how and why performance improvement happens. To shed light on these research questions, we theoretically investigate different training strategies for DR models and try to explain why hard negative sampling performs better than random sampling. Through the analysis, we also find that there are many potential risks in static hard negative sampling, which is employed by many existing training methods. Therefore, we propose two training strategies named a Stable Training Algorithm for dense Retrieval (STAR) and a query-side training Algorithm for Directly Optimizing Ranking pErformance (ADORE), respectively. STAR improves the stability of DR training process by introducing random negatives. ADORE replaces the widely-adopted static hard negative sampling method with a dynamic one to directly optimize the ranking performance. Experimental results on two publicly available retrieval benchmark datasets show that either strategy gains significant improvements over existing competitive baselines and a combination of them leads to the best performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7b577ba0e4230b2ac58d297b3d2cfc3d2f1aaace

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Boosted Dense Retriever (2021, background): “The success of Dense Retrieval has led to many recent papers proposing schemes to improve dense retriever training by in-novating on how negatives are sampled (Xiong et al., 2020; Qu et al., 2021; Zhan et al., 2021c; Lin et al., 2021, inter alia.)”
Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021, methodology): “However, they focus on other areas and do not consider the characteristics of DR in document retrieval, such as the importance of negative sampling [43, 47].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “, a sampler over the text collection based on some relevance model) is fixed or updated, hard negatives can be roughly divided into static hard negatives and dynamic hard negatives [98], [111].”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, methodology): “JPQ (Zhan et al., 2021a) and its extension, RepCONC (Zhan et al., 2021b), use product quantization (PQ) to compress embeddings, and jointly train the query encoder along with the centroids produced by PQ via a ranking-oriented loss function.”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, neutral): “, 2021), ADORE+STAR (Zhan et al., 2021) and PAIR (Ren et al.”
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, background): “DPR’s performance depends critically on the quality of negative examples [91, 92].”
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “Most modern information retrieval systems are based on the index-then-retrieve pipeline where documents are first encoded and stored in an inverted index [10, 13, 15, 26, 34, 55, 58] or a nearest neighbor search index [7, 14, 20, 21, 25, 29, 49, 51] and search results are then constructed based on a”
Task-aware Retrieval with Instructions (2022, background): “Negative samples are crucial for training retrieval systems (Zhan et al., 2021; Qu et al., 2021).”
The Power of Noise: Redefining Retrieval for RAG Systems (2024, positive): “For this purpose, instead of Contriever, we employed ADORE [46], a state-of-the-art retriever trained with “dynamic hard negatives”, to select the related documents a in our experimental setup.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “Sentence-window retrieval [175] is an efficient approach that enhances retrieval by fetching small chunks of text and returning a window of relevant sentences surrounding the retrieved segment.”
Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report (2025, unknown): “Subsequently, we move on to the next step of dense embedding pre-training: supervised fine-tuning on higher quality data, with mined hard negatives [37].”
ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025, unknown): “To further improve the quality of semantic matching, recent work has focused on refining this embedding space through contrastive learning with relevance supervision (Karpukhin et al., 2020; Zhan et al., 2021) or leveraging weakly supervised training signals (Xie et al., 2023).”