Towards Unsupervised Dense Information Retrieval with Contrastive Learning

By Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave. 2021

142 citations recorded by the source corpus.

Subfields: Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

1. Luyu Gao
2. Jamie Callan
3. Wen-tau Yih
4. Jimmy J. Lin
5. Danqi Chen
6. Jing Liu
7. Ruiyang Ren
8. Patrick Lewis

Citations in the corpus

• Galactica: A Large Language Model for Science (2022, methodology): “For dense retriever baselines, we evaluate two different Contriever models (Izacard et al., 2021).”
• Learning to Retrieve Passages without Supervision (2021, methodology): “In order to perform this operation efficiently at test time, a maximum-inner product search (MIPS) index (Johnson et al., 2021) is built over the encoded passages {EP (p1), ..., EP (pm)}.”
• Improving Passage Retrieval with Zero-Shot Question Generation (2022, methodology): “Contriever uses momentum contrastive training to learn dense retrievers from text paragraphs (Izacard et al., 2021).”
• ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “Specifically, one line of research focus on retrieval-augmented language models [58, 3, 17, 27, 30, 80, 34, 72], where they use sparse [54] or dense retrieval [22, 16] to extract relevant knowledge from the corpus.”
• Improving Text Embeddings with Large Language Models (2023, methodology): “To address this challenge, methods like Contriever [17], OpenAI Embeddings [29], E5 [44], and BGE [46] adopt a multi-stage training paradigm.”
• Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “This can be easily learned using unsupervised contrastive learning (Izacard et al., 2021; Gao et al., 2021; Gao and Callan, 2022).”
• Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “However, Contriever [28] shows that random cropping with data augmentation is more effective than ICT on a range of zero-shot information retrieval tasks.”
• Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “sults demonstrate that unsupervised contrastive pretraining leads to good performance in both zero-shot and few-shot settings [130].”
• MTEB: Massive Text Embedding Benchmark (2022, methodology): “(a) Transformer encoder methods coCondenser (Gao and Callan, 2021), Contriever (Izacard et al., 2021), LaBSE (Feng et al., 2020) and SimCSE-BERT-sup (Gao et al., 2021b) are based on the pre-trained BERT model (Devlin et al., 2018). coCondenser and Contriever add a selfsupervised stage prior to super”
• Text and Code Embeddings by Contrastive Pre-Training (2022, methodology): “…to note that previous embedding search methods do not report performance on sentence similarity tasks (Karpukhin et al., 2020a; Sachan et al., 2021; Izacard et al., 2021) and previous sentence embedding methods do not evaluate on search tasks (Reimers & Gurevych, 2019; Giorgi et al., 2020; Gao et…”
• Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2021, methodology): “When adopting a more advanced base retriever, such as Contriever (Izacard et al., 2021), SPAR achieves new state of the art with either Λ, highlighting the superior out-of-”
• Large Dual Encoders Are Generalizable Retrievers (2021, background): “(2021) on NDCG@10.”
• SGPT: GPT Sentence Embeddings for Semantic Search (2022, methodology): “Lastly, we highlight that on Quora SGPT-BE-5.”
• MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023, methodology): “Then, an off-the-shelf retrieval model (we use the pretrained Contriever model; Izacard et al. 2021) is used to embed all the edit statements and save them in a retrieval index.”
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “However, supervised methods suffer from the high cost of annotation, weak generalizability to new domains or languages, and restricted commercial use (Izacard et al., 2022a).”
• Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, methodology): “• Supervised adaptation models are trained with downstream task supervision, including RAG trained on BioASQ, Contriever fine-tuned on FiQA, and docT5query, ANCE, ColBERT, and Contriever fine-tuned on MS MARCO (Nogueira and Lin, 2019; Xiong et al., 2021; Khattab and Zaharia, 2020; Izacard et al., 20”
• HyQE: Ranking Contexts with Hypothetical Query Embeddings (2024, methodology): “To improve the zero-shot performance in unseen contexts, Contriever (Izac-ard et al., 2021) and other successive embedding models are trained via contrastive learning techniques.”
• LongEmbed: Extending Embedding Models for Long Context Retrieval (2024, methodology): “Among the 512-context models, E5 Base achieves the highest average score of 41.0 points, closely followed by E5-RoPE Base and Contriever.”
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval (2024, unknown): “…models have significantly advanced by exploiting supervision from natural language inference tasks and labeled query-document pairs, such as the MS-MARCO passage ranking dataset (Bajaj et al., 2016), to train effective text embeddings (Izacard et al., 2021; Wang et al., 2022; Xiao et al., 2024).”