Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
By Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, Arnold Overwijk. 2020
1,631 citations recorded by the source corpus.
Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval. Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval. In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing. This paper presents Approximate nearest neighbor Negative Contrastive Estimation (ANCE), a training mechanism that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus, which is parallelly updated with the learning process to select more realistic negative training instances. This fundamentally resolves the discrepancy between the data distribution used in the training and testing of DR. In our experiments, ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and sparse retrieval baselines. It nearly matches the accuracy of sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned representation space and provides almost 100x speed-up.
Subfields: Artificial Intelligence, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Hannaneh Hajishirzi
- Jimmy J. Lin
- Graham Neubig
- Percy Liang
- Sebastian Riedel
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Luyu Gao
- Sebastian Hofstätter
- Wen-tau Yih
- Sebastian Riedel
- Omar Khattab
- Gautier Izacard
Citations in the corpus
- Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “t relying on additional pretraining or complex joint training schemes. DPR has also been used as an important module in very recent work. For instance, extending the idea of leveraging hard negatives,Xiong et al. (2020a) use the retrieval model trained in the previous iteration to discover new negat”
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Popular supervised retrievers include DPR (Karpukhin et al., 2020), which is trained to discriminate the relevant passage among negative passages, and extensions such as ANCE (Xiong et al., 2020) which improved the hard negatives mining process.”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Xiong et al. (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS MARCO dataset.”
- Learning to Retrieve Passages without Supervision (2021, methodology): “Second, when fine-tuning on 128 examples only, we lower the batch size to 32 and run for 80 epochs.7 We use BM25 negatives produced by Karpukhin et al. (2020), and do not create hard negatives by the model itself (Xiong et al., 2021).”
- R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “…can be attributed to advances in neural retrieval methods (Karpukhin et al., 2020; Izacard and Grave, 2020; Khattab et al., 2020; Luan et al., 2021; Xiong et al., 2020, inter alia) that benefit from smarter negative sampling strategies or a better trade-off between complex question-passage…”
- Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “More sophisticated negative selection is proposed in concurrent work (Xiong et al., 2020).”
- Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “This training scheme has been successfully employed in different fields such as image and video classification [25], [53], [54], as well as natural language processing [55], [56] and audio representation learning [57], [58], [59].”
- Domain-matched Pre-training Tasks for Dense Retrieval (2021, methodology): “This procedure approximates the asynchronous model update which was shown to be helpful in Xiong et al. (2020).”
- Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, methodology): “The process of training these encoders can be quite complex, involving, for example, knowledge distillation [10, 16] and complex sampling techniques [26].”
- Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “Row (4b) shows results from ANCE [37], which appears to be more effective than TCT-ColBERT, although the comparison isn’t quite fair since our models were not trained on MS MARCO document data.”
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “While, hard negatives are shown to be important to train a dual-encoder (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020).”
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The irregular representation space raises the risk of degeneration [37, 55] and often necessitates sophisticated post-adjustment or fine-tuning to improve the sequence representations [16, 30, 32, 60].”
- From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (2021, background): “However, for self-supervised learning on graphs, it is non-trivial to construct informative negative examples since nodes are structurally connected, and selecting negative examples in an arbitrary manner may lead to large variance for stochastic gradients and slow training convergence [51].”
- CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2021, background): “Yu et al. (2021), Lin et al. (2021) and Kim and Kim (2022) attempt to fine-tune a dual encoder retriever (Xiong et al., 2021; Karpukhin et al., 2020) to handle conversational queries.”
- Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “10We also experimented with Approximate-nearest-neighbor Negative Contrastive Estimation (ANCE) [56]; however, it did not yield performance benefits.”
- Tool learning with large language models: a survey (2024, neutral): “In this example, GPT-4 selects the Metals Prices Rates API because it provides real-time updated information on gold prices.”